ESM hoists imports anyway but a mid-file `import` statement reads as a
foot-gun on review. No behaviour change; rules out one variable while
diagnosing a prod boot crash.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two complaints, both right:
1. QA sign-off was asking humans to OK trends + paid/organic without
anything visual to review — the dashboard wasn't built yet.
2. The QA panel just said "yes/no" and gave reviewers no specific
guidance on what they should be looking at.
Reordered the pipeline: after Stage 9 QA gates pass, immediately run
MoM compare (if prior_report_id) + Stage 10 Build, THEN park at
status=qa with the dashboard already rendered. Stage 10 has no Claude
cost (just file-copy + Vite build) so running it before sign-off costs
nothing meaningful, and the rationale "save spend if QA fails" never
actually applied — paid review work is already done by Stage 8.
QA panel rewrite:
- Big "Open dashboard ↗" button at the top
- Inline "What you're signing off" checklist split CM vs Strategist
(paid/organic + comments themes + sentiment risk on one side; trend
names + relevance + lens artefacts on the other)
- Removed the now-redundant Build button
- Second sign-off auto-completes the report; no extra click needed
- Status text walks them through the next action ("Open the dashboard,
walk through the views you're responsible for, then sign.")
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /cancel route used to 409 with "No running pipeline" when the child
handle wasn't in this server process — which happens any time the server
restarts (deploy, OOM, manual restart) while a run is still listed as
non-terminal in the DB. Three reports stuck "running for 11h+" on prod
this morning had no recoverable handle and no UI path to clear them.
Cancel is now smart enough to handle both cases:
- Live child: SIGTERM the process group + mark failed "Cancelled by ...".
- Orphan (no live child): mark failed "Marked failed by ... (no running
process — likely orphaned by a server restart)".
- Already terminal: 409 (unchanged).
Plus a boot sweep in server/index.ts that marks every non-terminal
report failed on startup with "Orphaned by server restart". This is the
right default — if the server is alive but the row is non-terminal, by
definition no child is producing artefacts for it. Saves the user
clicking Cancel on each stale row after every deploy.
Also adds v2/README.md with an architecture ASCII diagram, a 10-stage
pipeline ASCII diagram, the auth/tenancy model, ops commands, common
pitfalls (UK->GB, APIFY_LIVE_APPROVED, budget caps, cost-event
overwrite, compose-name policy), and the three deliberate design
choices behind V2's shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Dove2 run on prod failed because every hashtag/search seed 400'd
with "Field input.proxyCountryCode must be equal to one of the allowed
values" — Apify uses ISO codes ("GB"), not the colloquial "UK" stored
on the brief. Only profile scrapes (which don't pass proxyCountryCode)
got through, leaving 24 videos and a 16% manifest gate.
Two layers of fix:
- Brief Zod schema transforms geo: trims, uppercases, maps "UK" → "GB".
All briefs created or edited from now on are normalised at the form
boundary.
- Stage 2 also normalises at actor-input time, as belt-and-braces for
briefs already in the DB with "UK" written before this commit.
Plus a clear pre-flight error in Stage 8: when fewer than 5 videos made
it through analysis the trends schema literally can't be satisfied (each
trend needs ≥5 supporting_video_ids). Previously Claude tried, Zod
rejected with a 50-line "too_small" wall, and the operator was left
guessing. Now we throw a single sentence pointing at the actual cause:
the dataset is too small — adjust the brief and force re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The run page now has a Cancel button next to the status pill while a
pipeline is non-terminal. Clicking it (after a confirm prompt) hits
POST /api/reports/:id/cancel, which marks the row as failed with
"Cancelled by <email>" and SIGTERMs the spawned tsx child's whole
process group — so the Claude CLI subshell, ffmpeg, and Apify-poll
loops all stop together rather than orphaning.
Implementation notes:
- Replaced runningReportId with runningChild that holds the child handle
- Removed child.unref() so we can manage it. Pulled both the run and
retry spawn paths into a single spawnPipeline() helper to avoid drift
- Mark the row before sending the signal so the spawn-exit handler's
COALESCE doesn't overwrite "Cancelled by" with "killed by SIGTERM"
- Cancel from a different server process (e.g. after a server restart)
returns 409 with a hint that the child handle is no longer in scope —
the user can mark it failed manually if needed
Already-completed stages are preserved on disk via .state sentinels, so
"Cancel + edit brief + Force re-run" works without re-spending on
finished stages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The handlers handleListReportsForBrief and handleRetryReport were
defined in routes/reports.ts but never registered in the route table —
the brief detail page's Reports panel and the run page's Retry button
both 404'd against the deployed server with "Could not load reports:
Not found" / "Retry failed: Not found", even after a successful
deploy. Adding the route entries (and the missing imports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three bugs surfaced by the Dove2 demo run on prod and addressed together
because they all conspire to make the run page look dead:
1. Apify cost events were never persisted. Stage 2 imported onApifyCost
and registered an empty no-op callback inside its run loop, silently
overwriting the CLI's DB-writing handler set in cli.ts:logCost(). The
APIFY total stuck at $0.00 even though Stage 2 had spent $5+ in real
billing. Removed the override; the CLI's callback now wins.
2. Stage 4 inherited Stage 2's Pass-1 soft cap and skipped every actor.
resetBudget() sets a hard ceiling (95% of brief.budget_usd) and
setSoftCap() sets the Pass-1 cap (50%). Stage 2 fills the soft cap,
but Stage 4 never released it — every TIKTOK_TRANSCRIPTS / COMMENTS /
PROFILE call returned "budget reached — skipping" and the manifest
gate failed at 0% coverage. Stage 4 now calls setSoftCap(null) at
entry so it stays bounded only by the hard ceiling.
3. Even between cost events the run page had no liveness signal. Apify
actors run 1-3 minutes per scrape with no DB writes in flight, so the
UI looked frozen. Added a best-effort heartbeat: apify_client writes
.state/live_activity.json on every Apify status poll (every 5s),
GET /reports/:id includes it on the response, and the run page shows
a live banner with the current activity, elapsed time, last
heartbeat age (flags as suspicious past 90s), and running Apify
spend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 1000 likes / 10000 plays defaults are calibrated for top-of-funnel
beauty/fitness scrapes; in narrower TikTok niches almost every video lands
below them and Stage 2 returns 0 keepers. Defaults dropped to 100 likes /
1000 plays across:
- server/schemas/brief.ts (Zod default)
- db/init.sql (column default for new DBs)
- examples/dove-demo-brief.json
- operator-app's brief-form initial values
- operator-app's "Load Dove demo" inline brief
Stage 3 empty-pass1 error now reads pass1/spend_log.json and reports the
actual scrape breakdown — total $ spent, total raw videos returned, and
how many got dropped by each floor (zero-plays / min_plays / min_likes /
min_stl_pct). So instead of a generic "lower the floor", the user sees:
"Spent $5.42 across 7 scrapes; 1400 videos returned. Dropped: 12 zero-plays,
1305 below min_plays=10000, 31 below min_likes=1000."
Existing briefs are unaffected (column default applies to NEW rows). For the
in-flight Dove2 run the user can edit the brief and lower the floor, then
click Retry pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When you navigated away from a brief and came back, there was no indication
that a pipeline had already run for it — the page just showed the brief
fields and a "Run pipeline" button, making completed/in-flight runs invisible
without first hitting Home.
Now the brief detail page renders a "Reports for this brief" section listing
every run for the brief — status pill, run id, total cost, started/finished
relative timestamps, click-through to the run page. Auto-refreshes every 3s
while any run is non-terminal so an in-flight pipeline shows live progress
even when the user navigated to the brief instead of the report page.
Server:
- db/reports.ts: listReportsForBrief(brief_id, limit).
- routes/reports.ts: handleListReportsForBrief.
- index.ts: GET /api/briefs/:id/reports.
Client:
- api/reports.ts: useReportsForBrief hook with conditional polling.
- routes/briefs/detail.tsx: BriefReports section with status pills, in-flight
shortcut link, empty state when no runs exist yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues addressed:
1) Delete brief was awkward — only on the detail page, only for admin/owner,
and the on-disk briefs/<id>/ tree was orphaned even after a successful
DB delete. Server policy is now:
- team owner / admin: delete any brief in team
- team editor: delete only briefs they own
- team viewer: cannot delete
- super-admin: bypass
Plus: refuses to delete if a non-terminal pipeline run is active for the
brief, and rm -rf's briefs/<id>/ after the FK cascade so scrape data,
datasets and dashboards don't pile up.
Operator UI: per-row Delete button on the briefs list page (with confirm)
for users who have delete rights on each brief; detail-page rule loosened
to match the server's editor-owner policy.
2) Pipeline error messages were being clobbered. The server's child-on-exit
handler called finishReport() unconditionally, overwriting the actual
error the pipeline's main()-catch had just written. Now we use a
COALESCE update that only sets error_message when it's still NULL —
so the real failure (e.g. "Stage 3: pass1_videos.json is empty…") wins
over the generic "Pipeline exited with code 1". Stage 3's empty-pass1
error message is also now diagnostic: it lists the brief's current
engagement-floor values and suggests what to lower.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pipeline `all` command now skips stages whose .state/stage{N}.done
sentinel is present (unless --force), so retrying picks up exactly where
the previous run failed without re-spending on completed stages. Stage 5
(validate) is the deliberate exception — it always re-runs because
--drop-failing changes the selection set.
- pipeline/cli.ts: `all` wraps every stage in a maybeRun() helper that
checks the sentinel + writes one after running. Validate runs every
retry; if it doesn't reach 100% coverage the pipeline now fails LOUDLY
with a clear error rather than crashing in Stage 6.
- routes/reports.ts: handleRetryReport — POST /api/reports/:id/retry,
resets reports.status to pending (clears error_message + finished_at),
spawns `pipeline cli all` with --drop-failing (and --force when the
client passes {force:true}). Same singleton-running guard as run.
- operator-app: useRetryReport hook + two new buttons on Reports detail:
- "Retry pipeline" / "Force re-run" on the failure panel.
- "Retry with drop-failing" on the manifest panel (Stage 5 specific).
Items explicitly deferred (documented in head):
- SSE for live progress (3s React Query polling already in place).
- Conversational CLI brief intake (V3 §0; the operator-app form covers it).
62/62 unit tests pass. SPA build 284 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the last big spec gap: a single bundled React+Vite+Tailwind+Recharts
SPA that renders any report's dataset_v2.json under /api/reports/:id/dashboard/.
Spec deviation (intentional): rather than per-brief Vite builds (Netlify-
portable case), one bundle serves every report and fetches its dataset at
runtime. The portable HTML bundle (§10b) still ships per-report for offline
upload to claude.ai.
Dashboard template (v2/templates/dashboard_template):
- React 18 + Vite + TypeScript + Tailwind + Recharts. Same dark theme tokens
as the operator app.
- 9 views: Overview, Categories, Trends explorer (filters + sort + drilldown),
Lenses (4 sub-tabs for Hooks Library / Visual Vernacular / Audio Atlas /
Sentiment Map), Charts (engagement-vs-reach scatter, category treemap,
paid-vs-organic stacked bar), Compare (MoM new/returning/faded + category
momentum), Paid creators appendix, Methodology.
- Tab-based navigation with hash-stable URLs so deep links work.
Server:
- GET /api/reports/:id/dataset returns the on-disk dataset_v2.json (auth-
gated; 404 if Stage 10 hasn't built it yet).
- handleDashboardServe rewrite: serves (1) the explicit dashboard.html bundle,
(2) per-report on-disk static (covers, dataset_v2.json), (3) the bundled
SPA's assets/* and index.html with SPA fallback.
Build:
- v2/package.json workspaces re-includes templates/dashboard_template (was
excluded after earlier cutover-prep).
- Dockerfile.v2 builds the dashboard template alongside operator-app and
copies dist/ into the runtime image.
Local build: 838 modules transformed, 580 kB / 166 kB gzipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the V3 brief gaps that were called out in the audit.
Stage 8c lens artefacts (V3 §8c):
- prompts/lens_enrichment.md: rubric for Hooks Library / Visual Vernacular /
Audio Atlas / Sentiment Map.
- stages/stage_8c_lenses.ts: zod-validated lens generation from atomic
insights + per-video analyses; writes lenses/{hooks_library,visual_vernacular,
audio_atlas,sentiment_map}.json.
- stage_10_build.ts: dataset_v2.json now includes the four lens arrays.
- cli.ts: new `lenses` command; `all` runs 8c after Stage 8 (fail-soft);
`build` runs 8c too in case the previous run skipped it.
Pipeline split for sign-off enforcement (V3 §9):
- cli.ts `all` command now stops at status=qa after Stage 9. The operator app
drives Stage 10 separately via POST /api/reports/:id/build, which the
server only allows after CM + Strategist sign-offs from two different
humans.
- routes/reports.ts: handleQaSignoff (POST /api/reports/:id/qa/sign with
role=cm|strategist), handleBuildReport (verifies both signoffs + different
user_ids, then spawns `cli.ts build`). handleGetReport now also returns
manifest summary + qa.{cm_signoff,strategist_signoff}.
Brief edit (PATCH):
- db/briefs.ts: updateBrief.
- routes/briefs.ts: handleUpdateBrief, with editor+ team role.
- /api/briefs/:id PATCH route added.
- operator-app: useUpdateBrief hook; new /briefs/:id/edit route — minimal
JSON-textarea form, prefilled from brief.full, with Zod-issue surfacing.
- briefs/detail: "Edit" button next to Export/Run.
Reports detail UI:
- ManifestPanel: when manifest summary is in the response, render asset-
status grid + collapsible missing-videos list + the exact CLI command to
--drop-failing-backfill.
- SignoffPanel: two cards (CM + Strategist) showing signed-by-email/at;
"Sign as ..." button per side; client-side guard prevents the same user
signing both; "Build report" button enabled only when both signoffs
present + different humans.
- Dashboard static-serve route + Open dashboard / Download bundle from
earlier session re-confirmed wired.
Server clean, vite build green at 282 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the gap between "brief exists" and "report ships". The Phase A
placeholders for Home and Reports/detail are now real, and the brief detail
page can actually start a pipeline run.
Server (no schema changes — reports table already existed):
- db/reports.ts: createReport, getReport, getReportWithBrief, listReportsForTeam,
updateReportStatus, finishReport, logCostEvent (atomically updates the
reports row's running totals), listCostEvents.
- routes/reports.ts: GET /api/reports (active team), GET /api/reports/:id
(with cost_events), POST /api/briefs/:id/run that
1. authorises (editor+ on the brief's team),
2. creates a reports row (status=pending),
3. spawns the pipeline as a detached child running
`tsx pipeline/cli.ts all --report <brief_id> --run-id <reports.id>`,
4. returns the new report id.
Singleton flag prevents two concurrent runs (mirrors V1).
Pipeline:
- cli.ts: new --run-id flag. New `all` command drives every stage in order
via a withStage() helper that updates reports.status / current_stage at
each step. Cost callbacks now ALSO write to cost_events when run-id is
set, tagged with the current stage. main()'s catch handler calls
finishReport(runId, 'failed', err.message) so the UI doesn't poll forever
on a crash.
Client:
- api/reports.ts: useRecentReports, useReport (auto-polls every 3s while
status is non-terminal), useRunPipeline.
- routes/home.tsx: real recent-reports list — status pill, brief client +
business question, cost split, relative time.
- routes/reports/detail.tsx: full run page — header with status pill,
10-step pipeline progress with current-stage pulse, error block on
failure, three-tile cost summary (total / apify / claude), cost-event
log (most recent first, scrollable, sticky header), "Open dashboard"
+ "Download HTML bundle" actions when the run completes.
- routes/briefs/detail.tsx: Run pipeline button is now functional for
editors+, with a confirm dialog (warns about Apify/Claude spend),
navigates to the new /reports/:id on success, surfaces 409 if another
run is in flight.
62/62 unit tests still pass. Typecheck + vite build green; bundle 269 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V2 lives entirely under v2/ and is built around three asks the team raised
about V1: per-video assets sometimes drifted onto the wrong trend, hashtag
scrapes returned junk that wasn't filterable per-client, and there was no
multi-user model behind Microsoft SSO.
Highlights:
- Stable TikTok numeric-id key for every per-video asset; URL form drift is
logged loudly to drift_log.jsonl and never silently nulls assets. Stage 5
manifest hard-gates Stage 6 if any selected video is missing any required
asset; --drop-failing auto-backfills from the next-best recipe candidates.
- Per-brief engagement floor (min_likes / min_plays / min_stl_pct), applied
at Apify scrape time and re-validated locally; spend_log.json records
raw_returned vs kept_after_floor per scrape.
- Users + teams + memberships with owner/admin/editor/viewer roles; SSO
upserts a user keyed on Azure oid, auto-creates a personal team, and a
super-admin is bootstrapped via BOOTSTRAP_SUPER_ADMIN_EMAIL on first
sign-in. Phase A integration test: 16/16 pass.
- 10-stage TS pipeline (brief → seed → scrape1 → select → scrape2 →
validate → analyse → insights → trends → qa → build) wired through one
CLI; each stage idempotent + resumable from disk via .state sentinels.
§4.5 rubrics shipped under prompts/ and loaded into Claude calls.
- React 18 + Vite + TS + Tailwind operator SPA: brief intake form,
team management, super-admin user list, help/FAQ ported from V1.
- Separate Docker Compose project (name: social-reporting-v2, port 3457,
Postgres 5437) with deploy/setup-v2.sh, deploy-v2.sh, rollback-to-v1.sh
scripts that take over V1's /social-reports URL and let us roll back.
Verification: 62 unit tests pass (auth/session, ids extractor with full URL
fixture, engagement floor, recipes, manifest, linking-fix, MoM compare).
Live smoke run on a Dove brief: 1400 raw → 253 kept (82% culled) → 21
fully-bundled videos → 25 editorial trends across 8 brief-driven categories,
with drift=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
