marriott-box-image-video-tagging/README.md

Marriott Box Asset Tagger

Batch-processes images and videos in a Box folder, analyzes them with Gemini AI, and writes structured metadata back to Box using the marriottUsa metadata template. Videos use Box's 480p MP4 proxy representations to keep bandwidth and Gemini token usage manageable.

Every Gemini-analysed file is also written to a Postgres tagging_events table for search/audit, and there's a small React SPA on top (search across all logged fields, click through to the original Box file, trigger an on-demand tagging pass).

Components

• scheduler.py (tagger container) — APScheduler that fires main.main() on a cron (default daily 02:00).
• api.py (api container, FastAPI) — search, list runs, kick off ad-hoc runs in a background thread.
• db.py + schema.sql — Postgres logging layer.
• frontend/ (Vite + React + TS) — single-page UI, served by Apache from /var/www/html/marriott-tagging/ in prod. Auth via @azure/msal-react with a VITE_DEV_AUTH_BYPASS switch.

Setup

1. Clone and create virtual environment

cd Marriott_Box_Asset_Tagging
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

2. Box JWT credentials

Download your Box app's JWT config from the Box Developer Console and save it as box_config.json in the project root.

The service account must have:

• Access to folder 370595013246
• Permission to read/write metadata using the marriottUsa template

3. Gemini API key

Add your key to .env:

GEMINI_API_KEY=your_key_here

Get a key at Google AI Studio.

Usage

source env/bin/activate
python main.py

The script will:

1. Authenticate with Box and Gemini
2. Fetch the marriottUsa template schema (fields, types, allowed values)
3. Build a dynamic Gemini prompt from the schema
4. Recursively list all image and video files in the target folder
5. For each image: download, resize, analyze with Gemini, validate metadata, write metadata + description to Box
6. For each video: fetch the 480p MP4 proxy from Box, analyze with Gemini, write metadata + description + a scene-breakdown comment to Box
7. Print a summary of results

Run with Docker

Brings up Postgres, the scheduler (tagger), and the FastAPI backend (api). The frontend is built separately by deploy/deploy.sh (or npm run dev locally) and consumed by the API.

1. Prereqs

• Docker Desktop (or Docker Engine + Compose v2)
• box_config.json in the project root
• A .env copied from .env.example, filled in

cp .env.example .env
# minimum: GEMINI_API_KEY, POSTGRES_PASSWORD
# leave DEV_AUTH_BYPASS=true for now if you don't have Azure IDs ready

2. Build and start the backend services

docker compose up --build -d

This brings up three services:

• db — Postgres 16, named volume pgdata, port bound to 127.0.0.1:${POSTGRES_HOST_PORT:-5432}.
• tagger — runs scheduler.py (cron-driven Gemini passes).
• api — runs uvicorn api:app on container port 8000, published to 127.0.0.1:${MARRIOTT_API_PORT:-8004}.

3. Run the frontend (local dev)

cd frontend
npm install
npm run dev          # http://localhost:5173

Vite proxies /api/* to the FastAPI host port (default 8004). With VITE_DEV_AUTH_BYPASS=true you'll be auto-signed-in as the dev user.

4. Fire a tagging pass

Three ways:

• UI — open the SPA and click Run now. Polls live until done.
• API — curl -X POST http://127.0.0.1:8004/api/runs (DEV_AUTH_BYPASS=true) or with a Bearer token in prod.
• Container — docker compose exec tagger python main.py (bypasses the API entirely).

5. Inspect the DB

docker compose exec db psql -U marriott marriott_tagging -c '\d tagging_events'

docker compose exec db psql -U marriott marriott_tagging -c \
    "SELECT status, count(*) FROM tagging_events GROUP BY status;"

Auth: enabling Azure AD SSO

1. Register (or reuse) an Azure AD app. Redirect URI:
   • Local dev: http://localhost:5173
   • Prod: https://optical-dev.oliver.solutions/marriott-tagging/
2. Expose an API with scope access_as_user whose audience is the same client ID.
3. Fill .env:

   DEV_AUTH_BYPASS=false
   AZURE_TENANT_ID=...
   AZURE_CLIENT_ID=...
   VITE_DEV_AUTH_BYPASS=false
   VITE_AZURE_TENANT_ID=...
   VITE_AZURE_CLIENT_ID=...
   

4. docker compose up -d --force-recreate api and rebuild the SPA (deploy.sh does this on the server; locally cd frontend && npm run build).

Backend validates JWT signature against the tenant's JWKS, checks aud == AZURE_CLIENT_ID and iss matches one of the tenant URLs. With bypass=true, every request is logged as the DEV_AUTH_EMAIL user.

Stop / tear down

docker compose down            # stops containers, keeps the DB volume
docker compose down -v         # also deletes the DB volume (destroys data)

Notes

• Postgres failures never stop the tagger — db.log_event swallows errors. Box is the source of truth for "already tagged".
• The marriott-tagger.service / .timer files below remain for the older systemd deployment path; the Docker path is the recommended one. Don't run both on the same host.

Server Deployment (Docker — optical-dev.oliver.solutions)

This is the recommended path on the shared optical-dev.oliver.solutions dev server. Apps live under /opt/<slug>/ with an idempotent deploy/deploy.sh. Mirrors the OSOP / adeo split-build pattern: backend in Docker, SPA built and served by Apache from /var/www/html/marriott-tagging/.

Public URL: https://optical-dev.oliver.solutions/marriott-tagging/

First-time setup

# 1. Clone into /opt
sudo git clone git@bitbucket.org:zlalani/marriott-box-image-video-tagging.git \
    /opt/marriott-box-image-video-tagging
sudo chown -R "$USER:$USER" /opt/marriott-box-image-video-tagging
cd /opt/marriott-box-image-video-tagging

# 2. Drop credentials in place (NOT in git)
cp .env.example .env
$EDITOR .env                                    # GEMINI_API_KEY, POSTGRES_PASSWORD,
                                                # Azure IDs (or DEV_AUTH_BYPASS=true)
$EDITOR box_config.json                         # paste the Box JWT config

# 3. Deploy
./deploy/deploy.sh

The script will:

• Sanity-check .env, box_config.json, docker, git, compose v2.
• Pick free host ports — Postgres (default 5435, range 5435-5499) and API (default 8004, range 8003-8099) — persisted to .env.
• Render deploy/apache-marriott-tagging.conf from .tmpl with the picked API port.
• git pull --ff-only, docker compose build, docker compose up -d (db + tagger + api).
• Build the Vite SPA in a one-shot node:20 container; rsync frontend/dist/ to /var/www/html/marriott-tagging/.
• Poll /api/health until ready and verify the tagger container is running.
• Print the Apache Include line you need to add to the shared vhost.

One-time vhost step (manual): Edit /etc/apache2/sites-enabled/optical-dev.oliver.solutions.conf and add inside </VirtualHost>:

Include /opt/marriott-box-image-video-tagging/deploy/apache-marriott-tagging.conf

Then:

sudo apachectl configtest && sudo systemctl reload apache2

Re-deploying

cd /opt/marriott-box-image-video-tagging
./deploy/deploy.sh

Flags:

• --no-pull skip git pull
• --no-build skip docker compose build
• --no-frontend skip Vite build + SPA sync
• --run-now also fire a tagging pass via /api/runs (works with DEV_AUTH_BYPASS=true)
• --logs tail scheduler logs after deploy

Verifying it ran

# Scheduler logs (next cron-fired pass is at SCHEDULE_CRON; default 02:00 UTC)
docker compose logs -f tagger

# API logs
docker compose logs -f api

# Postgres request log
docker compose exec db psql -U marriott marriott_tagging -c \
    "SELECT status, count(*) FROM tagging_events GROUP BY status;"

Postgres is bound to 127.0.0.1 only — not reachable from outside the server. To inspect from your laptop, tunnel: ssh -L 55432:127.0.0.1:<POSTGRES_HOST_PORT> user@optical-dev.oliver.solutions, then psql postgresql://marriott:***@127.0.0.1:55432/marriott_tagging.

Notes

• The Docker deploy and the systemd deploy below target the same /opt/marriott-box-image-video-tagging/ directory. Pick one on any given server — don't run both, they'll both fire the tagger and double-write to Box.
• The SPA build bakes VITE_AZURE_* and VITE_DEV_AUTH_BYPASS into the bundle. Flipping the bypass requires a re-build (./deploy/deploy.sh does this).

Server Deployment (systemd, Ubuntu)

The repo includes marriott-tagger.service and marriott-tagger.timer for running the tagger as a scheduled service. These steps are written for Ubuntu 22.04 / 24.04 but should work on any systemd-based distribution with minor path tweaks (e.g. /sbin/nologin instead of /usr/sbin/nologin on Red Hat-family).

0. Prerequisites

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip

python3-venv is a separate apt package on Ubuntu — python3 -m venv will fail without it.

1. Clone the repo on the server

sudo mkdir -p /opt/marriott-box-image-video-tagging
sudo chown $USER:$USER /opt/marriott-box-image-video-tagging
git clone git@bitbucket.org:zlalani/marriott-box-image-video-tagging.git /opt/marriott-box-image-video-tagging
cd /opt/marriott-box-image-video-tagging

2. Create the service user

sudo useradd --system --shell /usr/sbin/nologin --home-dir /opt/marriott-box-image-video-tagging marriott-tagger
sudo chown -R marriott-tagger:marriott-tagger /opt/marriott-box-image-video-tagging

3. Drop credentials in place (NOT in git)

sudo -u marriott-tagger tee /opt/marriott-box-image-video-tagging/box_config.json > /dev/null < /path/to/local/box_config.json
sudo -u marriott-tagger tee /opt/marriott-box-image-video-tagging/.env > /dev/null <<'EOF'
GEMINI_API_KEY=your_key_here
EOF
sudo chmod 600 /opt/marriott-box-image-video-tagging/box_config.json /opt/marriott-box-image-video-tagging/.env

4. Set up the virtualenv

sudo -u marriott-tagger python3 -m venv /opt/marriott-box-image-video-tagging/env
sudo -u marriott-tagger /opt/marriott-box-image-video-tagging/env/bin/pip install -r /opt/marriott-box-image-video-tagging/requirements.txt

5. Install the systemd unit files

sudo cp /opt/marriott-box-image-video-tagging/marriott-tagger.service /etc/systemd/system/
sudo cp /opt/marriott-box-image-video-tagging/marriott-tagger.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now marriott-tagger.timer

6. Verify

# Show the next scheduled run
systemctl list-timers marriott-tagger.timer

# Trigger a one-off run immediately (timer will still run on schedule)
sudo systemctl start marriott-tagger.service

# Tail the logs (live)
sudo journalctl -u marriott-tagger -f

# Inspect the most recent run's full output
sudo journalctl -u marriott-tagger --since "1 day ago"

Updating the service

cd /opt/marriott-box-image-video-tagging
sudo -u marriott-tagger git pull
# If unit files changed:
sudo cp marriott-tagger.service marriott-tagger.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart marriott-tagger.timer

Configuration

Edit the constants at the top of main.py:

Setting	Default	Description
BOX_FOLDER_ID	(varies)	Box folder to process
METADATA_TEMPLATE_KEY	marriottUsa	Box metadata template key
GEMINI_MODEL	gemini-2.5-flash	Gemini model for analysis
EXCLUDED_FOLDER_PREFIXES	("z_", "zz_", "zzz_")	Subfolder name prefixes to skip
GEMINI_DELAY	7	Seconds between Gemini image calls
GEMINI_VIDEO_DELAY	10	Seconds between Gemini video calls
SKIP_ALREADY_TAGGED	True	Skip files with existing metadata
MAX_IMAGE_SIZE	1000	Max pixel dimension for image resize
VIDEO_SIZE_LIMIT_INLINE	20 MB	Below this, send video inline; above, use Gemini File API
VIDEO_SOURCE_SIZE_LIMIT	5 GB	Skip videos whose source file exceeds this
VIDEO_PROXY_SIZE_LIMIT	400 MB	Skip videos whose 480p proxy exceeds this (~60 min runtime)
MAX_FILES_PER_RUN	200	Hard cap on newly-tagged files per run
MAX_RUN_DURATION	4h	Hard wall-clock cap per run
DESCRIPTION_MAX_LENGTH	255	Box description field char limit

How It Works

• Dynamic prompt: The Gemini prompt is built at runtime from the actual Box template definition. If Marriott adds/changes fields or options in Box Admin, the script adapts automatically.
• Metadata + description: Each file gets structured metadata (for filtered search) and a short description (visible in Box list views, also indexed by Box search).
• Search-keyword tail: Each description is formatted as <summary sentence>. <comma-separated keywords>. — the keyword tail covers synonyms and broader category terms (e.g. food/dining/eating/meal/restaurant) so a search for "Food" hits assets tagged with the enum value Dining, etc.
• Video scene breakdown: Videos additionally get a timestamped scene breakdown written as a comment on the Box file — a high-level chapter map for finding moments inside long videos.
• Resumable: Files with existing metadata are skipped by default, so the script can be re-run after interruptions or when new files are added.
• Validation: Gemini output is validated against the template schema — invalid enum values are dropped, multiSelect arrays are filtered to allowed options only.
• Large-video gating: Videos exceeding the source or proxy size limits are skipped cleanly rather than wasting time / API budget on content beyond Gemini's context window. Skips are reported in the summary as skipped, not errored.
• Per-run limiter: A daily run will tag at most MAX_FILES_PER_RUN newly-tagged files in MAX_RUN_DURATION of wall clock. Whichever cap hits first, the run exits cleanly with a summary line; the next scheduled run picks up the remaining untagged files. This keeps a sudden 1000-file upload from blowing through your Gemini budget in one night.