enable mp4 download, smart Stage 4 cache, default Apify live in prod
Closes the last gaps so the operator app is end-to-end usable in production.
Server:
- routes/reports.ts: GET /api/reports/:id/dashboard[/<file>] serves files
out of the report's brief outputs/ tree (HTML bundle, dataset_v2.json,
any covers referenced relatively). Auth-gated by team viewer role.
Path-traversal guarded.
- index.ts: two new route patterns (with and without trailing path).
Client:
- routes/reports/detail.tsx: "Open dashboard" is a target=_blank anchor at
/api/reports/:id/dashboard/, "Download" same URL with download attribute.
No more dead SPA-internal link.
Pipeline polish (the four open items from the smoke test):
- stage_10_build.ts: covers are now downscaled via ffmpeg (240px / q=6)
before base64 inlining. Hard ceiling per cover 60 KB; falls back to the
original only if it already fits. Honours V3 brief's ≤3 MB HTML bundle.
- lib/apify_client.ts: post-run cost is re-polled with backoff (0/5/15/30s)
instead of a single read. TIKTOK_COMMENTS reports $0 immediately and
$5+ later — without this the soft cap can't fire on it.
- stage_2_pass1_scrape.ts: shouldDownloadVideos:true (and shouldDownloadCovers:true)
by default so videoMeta.downloadAddr is populated for Stage 4 frame
extraction. Disable with DISABLE_VIDEO_DOWNLOADS=true if the budget is
tight.
- stage_4_pass2_enrich.ts: Stage 5 backfill candidates aren't in the
transcripts/comments cache. New loadOrFetchActor() reads what's cached,
identifies missing ids, fetches just those from Apify, and merges back
into the cache. Backfill no longer drops every candidate.
Production defaults:
- .env.example: APIFY_LIVE_APPROVED=true (commented; operators can flip
to false for dry-runs).
- cutover-in-place.sh: sets APIFY_LIVE_APPROVED=true if not already in .env
after the migration step, so a fresh prod cutover doesn't accidentally
dry-run.
62/62 unit tests pass; tsc + vite build green; bundle 269 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the gap between "brief exists" and "report ships". The Phase A
placeholders for Home and Reports/detail are now real, and the brief detail
page can actually start a pipeline run.
Server (no schema changes — reports table already existed):
- db/reports.ts: createReport, getReport, getReportWithBrief, listReportsForTeam,
updateReportStatus, finishReport, logCostEvent (atomically updates the
reports row's running totals), listCostEvents.
- routes/reports.ts: GET /api/reports (active team), GET /api/reports/:id
(with cost_events), POST /api/briefs/:id/run that
1. authorises (editor+ on the brief's team),
2. creates a reports row (status=pending),
3. spawns the pipeline as a detached child running
`tsx pipeline/cli.ts all --report <brief_id> --run-id <reports.id>`,
4. returns the new report id.
Singleton flag prevents two concurrent runs (mirrors V1).
Pipeline:
- cli.ts: new --run-id flag. New `all` command drives every stage in order
via a withStage() helper that updates reports.status / current_stage at
each step. Cost callbacks now ALSO write to cost_events when run-id is
set, tagged with the current stage. main()'s catch handler calls
finishReport(runId, 'failed', err.message) so the UI doesn't poll forever
on a crash.
Client:
- api/reports.ts: useRecentReports, useReport (auto-polls every 3s while
status is non-terminal), useRunPipeline.
- routes/home.tsx: real recent-reports list — status pill, brief client +
business question, cost split, relative time.
- routes/reports/detail.tsx: full run page — header with status pill,
10-step pipeline progress with current-stage pulse, error block on
failure, three-tile cost summary (total / apify / claude), cost-event
log (most recent first, scrollable, sticky header), "Open dashboard"
+ "Download HTML bundle" actions when the run completes.
- routes/briefs/detail.tsx: Run pipeline button is now functional for
editors+, with a confirm dialog (warns about Apify/Claude spend),
navigates to the new /reports/:id on success, surfaces 409 if another
run is in flight.
62/62 unit tests still pass. Typecheck + vite build green; bundle 269 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the three Phase A scaffold placeholders with working pages backed by
the existing server endpoints (no server changes).
- src/api/teams.ts (new): useTeams, useTeam, useCreateTeam, useAddMember,
useUpdateMemberRole, useRemoveMember. All with cache invalidation.
- src/api/admin.ts (new): useAllUsers, useToggleSuperAdmin.
- routes/teams/list.tsx: list of teams (cards with role badge + slug + Personal
marker), inline "Create a team" form (POSTs /api/teams), each card links to
/teams/:id. Inline 409 / validation handling.
- routes/teams/detail.tsx: team header with my role; members table; owners can
change member roles via dropdown, owners + admins can remove members
(confirm dialog); below the table, an Invite form (email + role select)
matching POST /api/teams/:id/members. Per-row + per-form error surfacing.
- routes/admin/users.tsx: full users table — email, name, super-admin badge,
created/last-login timestamps, promote/revoke button. Disables the toggle
for the current user when they're super-admin (matches the server's
self-demotion guard); also shows "(you)" indicator next to your own row.
Bundle size: 258 kB → /social-reports/assets/index-C0ofQc9Y.js (+7 kB gzipped).
TS strict + vite build pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The server returns { user, teams, active_team } (server/routes/me.ts:19-30).
The SPA's MeResponse type had `memberships`, so TeamSwitcher's `data.memberships.length`
crashed on initial render after sign-in:
TypeError: Cannot read properties of undefined (reading 'length')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues from the first server cutover:
1. SPA loaded white-on-black blank because import.meta.env.VITE_AZURE_TENANT_ID
and VITE_AZURE_CLIENT_ID were undefined at runtime. Vite reads VITE_* at
*build time* and inlines them into the bundle; passing them only as
runtime container env vars is too late.
- Dockerfile.v2: declare ARG VITE_AZURE_TENANT_ID, VITE_AZURE_CLIENT_ID,
VITE_BASE; export as ENV before `npm run build`.
- docker-compose.v2.yml: forward AZURE_TENANT_ID / AZURE_CLIENT_ID /
VITE_BASE through `build.args` so the cutover .env values reach Vite.
2. cutover-in-place.sh stopped V1 with `-p social-listening`, but V1's actual
compose project name was `social-reporting` (parent dir). Old V1 containers
were left running. Now we try both project names AND sweep by container
name pattern (anything matching social-listening or social-reporting-db-1
that isn't a V2 container).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The host port 5437 on the optical-dev server was already allocated by
something (probably an old stopped-but-not-removed Postgres container
or another tracking app). V2 doesn't need a host port for db-v2 in
production — app-v2 reaches it over the docker network at db-v2:5432.
Per CLAUDE.md "always check for ports that are already used":
- docker-compose.v2.yml: remove the unconditional db-v2 host port
binding. Compose's list-merge semantics meant `ports: []` in the prod
override didn't actually clear the base list.
- docker-compose.v2.dev.yml (new): local-dev overlay that re-adds the
host port for psql convenience. Use with `-f base -f dev`. Bound to
127.0.0.1 so the db is never reachable from outside the dev machine.
- cutover-in-place.sh: pre-flight check on APP_V2_PORT (3457) — if
it's held by something other than our own V2 container, abort with
a clear message rather than failing mid-deploy.
Verified locally: `compose -f base -f prod up -d` brings up a stack
with db-v2 having no host port (just internal 5432/tcp), app-v2 on
127.0.0.1:3457, /api/health returns {ok:true}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The unanchored `briefs/` rule (intended for the runtime per-report tree at
the repo root) also matched `v2/operator-app/src/routes/briefs/` — so the
brief-form components never made it into git, and the prod docker build
failed at vite with "Could not resolve ./routes/briefs/list".
Fix: anchor to the two specific paths that should be ignored, /briefs/
and /v2/briefs/. The React routes dir under src/ is now tracked.
Adds the four missing files: list.tsx, new.tsx, detail.tsx, _form-bits.tsx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A scaffolded the SPA at the bare origin (`/`); production lives behind
Apache at `/social-reports/`. Without these fixes, V2's built assets 404 and
Azure SSO rejects the redirect URI mismatch.
- Vite `base: /social-reports/` (overridable via VITE_BASE for dev).
- BrowserRouter basename = import.meta.env.BASE_URL.
- apiFetch + msal-browser script src + token-exchange URL all prefix BASE.
- MSAL redirectUri now matches V1's Azure-registered URI:
`${origin}/social-reports/login.html`.
- New `<Route path="/login.html">` alias renders the same Login component
so React Router matches the redirect URI when MSAL returns.
Deploy ergonomics (the user wants V1 gone from the server):
- v2/deploy/cutover-in-place.sh: run from /opt/social-reporting; stops V1,
pulls main (v2/ appears, V1 dirs deleted), migrates secrets from V1's
.env into v2/.env, swaps Apache, starts V2. Single command, no clone of
a sibling dir needed.
- setup-v2.sh: PURGE_V1=true flag now cleans /opt/social-reporting and
the V1 docker volume after V2 is healthy.
- rollback-to-v1.sh: re-clones the v1-archive branch when V1 is no longer
on disk (REPO_URL required).
62/62 unit tests still pass; vite build emits assets under /social-reports/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V1's running deployment at /opt/social-reporting on the server stays put
until cutover; V1's source is preserved on the v1-archive branch and via
git history. From this commit forward, all work targets v2/.
The new root README points contributors at v2/ and documents the rollback
path (deploy/rollback-to-v1.sh) for the cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V2 lives entirely under v2/ and is built around three asks the team raised
about V1: per-video assets sometimes drifted onto the wrong trend, hashtag
scrapes returned junk that wasn't filterable per-client, and there was no
multi-user model behind Microsoft SSO.
Highlights:
- Stable TikTok numeric-id key for every per-video asset; URL form drift is
logged loudly to drift_log.jsonl and never silently nulls assets. Stage 5
manifest hard-gates Stage 6 if any selected video is missing any required
asset; --drop-failing auto-backfills from the next-best recipe candidates.
- Per-brief engagement floor (min_likes / min_plays / min_stl_pct), applied
at Apify scrape time and re-validated locally; spend_log.json records
raw_returned vs kept_after_floor per scrape.
- Users + teams + memberships with owner/admin/editor/viewer roles; SSO
upserts a user keyed on Azure oid, auto-creates a personal team, and a
super-admin is bootstrapped via BOOTSTRAP_SUPER_ADMIN_EMAIL on first
sign-in. Phase A integration test: 16/16 pass.
- 10-stage TS pipeline (brief → seed → scrape1 → select → scrape2 →
validate → analyse → insights → trends → qa → build) wired through one
CLI; each stage idempotent + resumable from disk via .state sentinels.
§4.5 rubrics shipped under prompts/ and loaded into Claude calls.
- React 18 + Vite + TS + Tailwind operator SPA: brief intake form,
team management, super-admin user list, help/FAQ ported from V1.
- Separate Docker Compose project (name: social-reporting-v2, port 3457,
Postgres 5437) with deploy/setup-v2.sh, deploy-v2.sh, rollback-to-v1.sh
scripts that take over V1's /social-reports URL and let us roll back.
Verification: 62 unit tests pass (auth/session, ids extractor with full URL
fixture, engagement floor, recipes, manifest, linking-fix, MoM compare).
Live smoke run on a Dove brief: 1400 raw → 253 kept (82% culled) → 21
fully-bundled videos → 25 editorial trends across 8 brief-driven categories,
with drift=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace cp frontend/* with cp -r frontend/. to copy all files reliably
- Add mkdir -p as safety net in deploy.sh
- Add apache2 reload after frontend copy in deploy.sh
- setup.sh now copies entire frontend dir instead of hardcoded filenames
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Optional textarea lets users provide strategic guidance like objectives,
competitive context, and focus areas. Injected into Claude prompts at
stages 2, 4, 6, and 8 so all agents can produce more focused output.
Backward compatible — empty context changes nothing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add apifyBudget field to ClientBrief (default $10)
- Budget split: 70% discovery (evenly across platforms), 30% enrichment
- Per-platform soft cap prevents one platform hogging the budget
- Budget input field added to both frontend and dashboard forms
- Saved briefs preserve budget setting
- Fix Claude Vision 5MB limit: filter oversized thumbnails before batching
- Fix Docker: ensure node user can write to volume-mounted dirs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix TikTok comments actor input: `videos` → `videoUrls` (wrong field name)
- Fix TikTok transcripts actor input: `videos` → `videoUrls` (wrong field name)
- Allow HTTP URLs for thumbnails (TikTok CDN uses HTTP)
- Add date filtering to profile scrapers (TikTok + Instagram)
- Keep videos with unparseable dates instead of dropping them
- Lower visual language threshold from 5 to 3 thumbnails
- Increase thumbnail timeout from 5s to 10s
- Add logging for failed thumbnail downloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Remove Desk Research (Stage 7 skipped, sources removed from report)
2. Fix comments scraping: increase cap to 2000, handle alt field names
3. Dynamic stats bar: hide zero-value stats instead of showing "0 Comments"
4. Prompt improvements: enforce timeliness, comment-based insights, creator spotlight algorithm (2-10 videos, exclude >50% dominance)
5. Date filtering: pass date params to Apify actors (oldestCreateTime, onlyPostsNewerThan, uploadDate) + log filter counts
6. Pullquotes: 3-4 generated editorial dividers between sections
7. Thumbnails: download top 50 coverUrl as base64, store on EnrichedVideo
8. Visual Language section: 5 batches of 10 through Claude Vision, synthesized into 5-6 visual codes with thumbnail cards
9. Sticky navigation bar with anchor links to all sections
10. New types: VisualCode, thumbnailUrl on Video, thumbnailBase64 on EnrichedVideo, pullquotes/visualCodes on ReportJSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fixed null reference error when loading JSON files (removed deleted jsonPreview element refs)
- Added Export button to download saved briefs as JSON files
- Renamed "Load & Run" to just "Load" per user feedback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Backend: GET/POST/DELETE /api/briefs endpoints storing JSON files in briefs/ dir
- Frontend: new Saved Briefs tab with cards showing client details, Load & Run, Delete
- Save Current Brief button on Pipeline tab persists form to server
- Both standalone dashboard and static frontend updated
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete individual runs (with confirmation)
- Bulk remove all failed or completed runs
- Download report as HTML file (Content-Disposition: attachment)
- View + Download buttons in history table
- Backend: DELETE /api/runs/:id and DELETE /api/runs?status=failed|completed
- Backend: GET /report/:id/download serves with attachment header
- Updated both frontend/index.html and dashboard/index.html
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ProxyTimeout is not allowed in <Location> context. Moved to server-level
ProxyTimeout directive already set above.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EventSource auto-reconnects on connection drop, which re-fires the
'connected' event. The handler was POSTing /run on every reconnect,
causing multiple parallel pipeline runs and runaway Apify costs.
Added pipelineStarted guard so /run only fires on first connect.
Fixed in both frontend/index.html and dashboard/index.html.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- TikTok transcripts/comments actor expects 'videos' not 'videoUrls'
- Wrap all enrichment actor calls in safeRunActor so failures skip
instead of crashing the pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Promise.all() launched all platform scrapers simultaneously, so multiple
expensive runs started before any costs were tracked. Budget check only
saw totals after each run finished, allowing $7+ overspend on a $5 limit.
Now Stage 3 and Stage 5 run each scraper sequentially so the budget
gate can cut off between calls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DB_PORT defaults to 5436, DASHBOARD_PORT defaults to 3456.
Prod override no longer redeclares ports (was causing duplicates).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Static frontend (index.html, login.html, config.js) for Apache serving
- JSON-based auth API endpoints (/api/login, /api/auth, /api/logout)
- Apache config with ProxyPass for /social-reports path
- deploy/setup.sh for Ubuntu + Apache + Docker deployment
- docker-compose.prod.yml binds ports to 127.0.0.1 only
- Configurable API base URL via frontend/config.js
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cookie-based session auth with login page (DASH_USER/DASH_PASS env vars)
- Serve generated reports via /report/:id route with View Report button
- YouTube iframe and Instagram native embeds in HTML reports
- Supporting videos grid per trend with platform icons
- Logout link in dashboard header
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
8-stage TypeScript pipeline with Apify scraping, Claude AI analysis,
real-time dashboard with SSE, PostgreSQL cost tracking, and Apify budget controls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
