ESM hoists imports anyway but a mid-file `import` statement reads as a
foot-gun on review. No behaviour change; rules out one variable while
diagnosing a prod boot crash.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two complaints, both right:
1. QA sign-off was asking humans to OK trends + paid/organic without
anything visual to review — the dashboard wasn't built yet.
2. The QA panel just said "yes/no" and gave reviewers no specific
guidance on what they should be looking at.
Reordered the pipeline: after Stage 9 QA gates pass, immediately run
MoM compare (if prior_report_id) + Stage 10 Build, THEN park at
status=qa with the dashboard already rendered. Stage 10 has no Claude
cost (just file-copy + Vite build) so running it before sign-off costs
nothing meaningful, and the rationale "save spend if QA fails" never
actually applied — paid review work is already done by Stage 8.
QA panel rewrite:
- Big "Open dashboard ↗" button at the top
- Inline "What you're signing off" checklist split CM vs Strategist
(paid/organic + comments themes + sentiment risk on one side; trend
names + relevance + lens artefacts on the other)
- Removed the now-redundant Build button
- Second sign-off auto-completes the report; no extra click needed
- Status text walks them through the next action ("Open the dashboard,
walk through the views you're responsible for, then sign.")
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude reliably generates 35-50 trends but ~half typically have a long
tail of 1-4 supporting videos — "patterns I noticed but can't strongly
back". The strict Zod min(5) on supporting_video_ids rejected the
ENTIRE response on the first such trend, throwing away the 20+ genuinely
strong trends in the same call. The Dove run on prod just hit this:
49 trends generated, 27 with <5, whole batch lost.
Fix:
- RAW_TRENDS_SCHEMA no longer enforces the supporting-videos minimum at
parse time. Lenient parse keeps the response intact.
- After parse, filter to ≥MIN_SUPPORTING_VIDEOS_PER_TREND (default 5,
env-overridable for small corpora).
- Dropped tail is logged to qa/dropped_low_support_trends.json for
forensics — shows what Claude noticed but couldn't strongly support.
- Hard fail only if ALL trends drop, with a clear remedy in the message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix only handled the candidates-exhausted exit path; the
maxBackfillRounds=3 cap was a separate exit and still left failing
videos in the selection. The user got 95.74% coverage with 2 failing
stragglers — every other id was clean, but the gate refused to pass.
After the loop, if dropFailing is on and the manifest still has
failing videos (because we capped on rounds, not candidates), trim
them out and rebuild. The contract on --drop-failing is "produce the
best run we can", not "100% or fail forever". Stage 8b's ≥5 minimum
is enforced separately for the truly-too-small case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Dove run hit 51% manifest coverage with 17 transcripts + 9 bundles
missing. Three real bugs uncovered in sequence by the manifest jq
introspection:
1. Transcripts gated as required. The Apify TIKTOK_TRANSCRIPTS actor
only returns text for videos with captions/subtitles; haircare,
beauty, dance, ASMR, and lifestyle TikTok content (which IS the
target audience for this kind of brief) often has none. Stage 6's
prompt already handles bundle.transcript === null gracefully — it
logs "(no transcript)" and Claude analyses on caption + 30 comments
+ frame note. Made transcripts advisory by default; opt back into
strict via MANIFEST_TRANSCRIPT_REQUIRED=true.
2. Backfill-exhaustion path didn't rebuild the manifest. When Stage 5
ran out of pass1 candidates to backfill with, it trimmed
selected_video_ids to survivors and broke the loop — but never
re-evaluated the manifest against the trimmed list. coverage_pct
stayed at the pre-trim value (e.g. 51%) and the gate failed even
though every remaining id was 100% bundled. Now rebuilds.
3. Added a fast-fail when fewer than 5 videos pass the gate. Stage 8b
trends require ≥5 supporting videos per trend; running Stage 6 + 7
on 3 passing videos burns Claude budget for an unsatisfiable
downstream schema. The cli error now distinguishes "coverage too
low" from "passing-set too small" and points at the right remedy
(lower min_likes/min_plays vs. raise min_plays to filter junk).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apify's mp4 download URLs have a documented 14-day TTL but in practice
expire much faster (often <24h). ffmpeg fails on stale URLs and the
manifest gate then fails at 50-80% coverage citing "missing frames" —
even though Stage 6's prompt only references frame count and Claude is
fine without them. Transcript + comments + cover + metadata are the
real analysis substrate; frames are visual context, nice-to-have.
Inverted the env knob: MANIFEST_FRAMES_REQUIRED=true to opt back into
the strict gate, default is now advisory. The cli error message also
now points operators at the actual signal — if the missing list is
mostly transcripts/comments, that's a seed pool / engagement floor
problem, not a frame extraction problem.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /cancel route used to 409 with "No running pipeline" when the child
handle wasn't in this server process — which happens any time the server
restarts (deploy, OOM, manual restart) while a run is still listed as
non-terminal in the DB. Three reports stuck "running for 11h+" on prod
this morning had no recoverable handle and no UI path to clear them.
Cancel is now smart enough to handle both cases:
- Live child: SIGTERM the process group + mark failed "Cancelled by ...".
- Orphan (no live child): mark failed "Marked failed by ... (no running
process — likely orphaned by a server restart)".
- Already terminal: 409 (unchanged).
Plus a boot sweep in server/index.ts that marks every non-terminal
report failed on startup with "Orphaned by server restart". This is the
right default — if the server is alive but the row is non-terminal, by
definition no child is producing artefacts for it. Saves the user
clicking Cancel on each stale row after every deploy.
Also adds v2/README.md with an architecture ASCII diagram, a 10-stage
pipeline ASCII diagram, the auth/tenancy model, ops commands, common
pitfalls (UK->GB, APIFY_LIVE_APPROVED, budget caps, cost-event
overwrite, compose-name policy), and the three deliberate design
choices behind V2's shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Dove2 run on prod failed because every hashtag/search seed 400'd
with "Field input.proxyCountryCode must be equal to one of the allowed
values" — Apify uses ISO codes ("GB"), not the colloquial "UK" stored
on the brief. Only profile scrapes (which don't pass proxyCountryCode)
got through, leaving 24 videos and a 16% manifest gate.
Two layers of fix:
- Brief Zod schema transforms geo: trims, uppercases, maps "UK" → "GB".
All briefs created or edited from now on are normalised at the form
boundary.
- Stage 2 also normalises at actor-input time, as belt-and-braces for
briefs already in the DB with "UK" written before this commit.
Plus a clear pre-flight error in Stage 8: when fewer than 5 videos made
it through analysis the trends schema literally can't be satisfied (each
trend needs ≥5 supporting_video_ids). Previously Claude tried, Zod
rejected with a 50-line "too_small" wall, and the operator was left
guessing. Now we throw a single sentence pointing at the actual cause:
the dataset is too small — adjust the brief and force re-run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The run page now has a Cancel button next to the status pill while a
pipeline is non-terminal. Clicking it (after a confirm prompt) hits
POST /api/reports/:id/cancel, which marks the row as failed with
"Cancelled by <email>" and SIGTERMs the spawned tsx child's whole
process group — so the Claude CLI subshell, ffmpeg, and Apify-poll
loops all stop together rather than orphaning.
Implementation notes:
- Replaced runningReportId with runningChild that holds the child handle
- Removed child.unref() so we can manage it. Pulled both the run and
retry spawn paths into a single spawnPipeline() helper to avoid drift
- Mark the row before sending the signal so the spawn-exit handler's
COALESCE doesn't overwrite "Cancelled by" with "killed by SIGTERM"
- Cancel from a different server process (e.g. after a server restart)
returns 409 with a hint that the child handle is no longer in scope —
the user can mark it failed manually if needed
Already-completed stages are preserved on disk via .state sentinels, so
"Cancel + edit brief + Force re-run" works without re-spending on
finished stages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage 2 was sequential: 6-12 hashtag/profile/search actors at 90-180s
each meant 10-25 minutes wall-time per pass-1 scrape. Now runs 4 actors
in parallel via a worker pool that re-checks the budget cap before
picking each new job. Tradeoff: up to 3 actors may complete after the
soft cap is hit and overshoot it slightly (~$3 worst case on a $25
cap), worth it to halve+ the wall time.
Stage 6 (per-video Claude analysis) bumped 4→8 — sits well inside
typical Anthropic tier-1 concurrency. Stage 4 (per-video frame
extraction + translate + bundle) bumped 4→8 — CPU/disk-bound but
well under the container's headroom. Both env-overridable
(STAGE6_CONCURRENCY, STAGE4_CONCURRENCY) for tight quotas.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Banner was showing only the Apify-side running cost from the heartbeat
file, so Stage 1 Claude spend ($0.06+) showed as $0.00 and Claude-only
stages (6/7/8) wouldn't show any spend at all. Now reads total cost
straight from the report row (apify + claude broken out as a sub-line)
and refreshes the heartbeat on every Claude cost event so the banner
stays alive between Apify scrapes too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The handlers handleListReportsForBrief and handleRetryReport were
defined in routes/reports.ts but never registered in the route table —
the brief detail page's Reports panel and the run page's Retry button
both 404'd against the deployed server with "Could not load reports:
Not found" / "Retry failed: Not found", even after a successful
deploy. Adding the route entries (and the missing imports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous BACKEND_DIR_V2=/opt/social-reporting-v2 hardcode failed on
the production host because we did an in-place cutover: V1 and V2 share
a single checkout at /opt/social-reporting. ./v2/deploy/deploy-v2.sh
exited immediately with "V2 dir not found".
Both deploy-v2.sh and rollback-to-v1.sh now derive their repo root from
the script's own location, so they work whatever the parent directory
is named. setup-v2.sh is left alone — it's the first-time provisioner
and creating /opt/social-reporting-v2 from scratch is still a fine
default for greenfield installs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three bugs surfaced by the Dove2 demo run on prod and addressed together
because they all conspire to make the run page look dead:
1. Apify cost events were never persisted. Stage 2 imported onApifyCost
and registered an empty no-op callback inside its run loop, silently
overwriting the CLI's DB-writing handler set in cli.ts:logCost(). The
APIFY total stuck at $0.00 even though Stage 2 had spent $5+ in real
billing. Removed the override; the CLI's callback now wins.
2. Stage 4 inherited Stage 2's Pass-1 soft cap and skipped every actor.
resetBudget() sets a hard ceiling (95% of brief.budget_usd) and
setSoftCap() sets the Pass-1 cap (50%). Stage 2 fills the soft cap,
but Stage 4 never released it — every TIKTOK_TRANSCRIPTS / COMMENTS /
PROFILE call returned "budget reached — skipping" and the manifest
gate failed at 0% coverage. Stage 4 now calls setSoftCap(null) at
entry so it stays bounded only by the hard ceiling.
3. Even between cost events the run page had no liveness signal. Apify
actors run 1-3 minutes per scrape with no DB writes in flight, so the
UI looked frozen. Added a best-effort heartbeat: apify_client writes
.state/live_activity.json on every Apify status poll (every 5s),
GET /reports/:id includes it on the response, and the run page shows
a live banner with the current activity, elapsed time, last
heartbeat age (flags as suspicious past 90s), and running Apify
spend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage 2 hashtag scrapes can take 1-3 min each, and the run page gave no
sign of life between cost events — the active stage just had a pulsing
dot. Now shows the elapsed run time in the header, a 'working…' marker
on the active stage, the latest cost event under it (e.g. 'apify ·
hashtag:#hairtok · 30s ago'), and a hint that long Apify silences are
mid-flight scrapes, not stalls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Spent $0.00 across 0 scrapes" diagnostic was technically correct but
hid the actual cause when every Apify actor call threw before returning.
The most common case in the wild — a missing or invalid APIFY_TOKEN — left
no useful trail in the UI.
Stage 2 now:
- Tracks actor exceptions per scrape into an actorErrors[] array.
- If spend log ends up empty AND any errors were captured, throws with the
first 3 errors inline so the operator sees the actual reason in the UI:
"every Apify call (7/7 attempted) threw before returning data … hashtag:#hairtok:
Apify start failed: 401 Unauthorized … Fix the token in v2/.env, restart V2,
click Force re-run."
- Persists actor_errors into spend_log.json for forensic inspection too.
BriefReports panel:
- Now shows the actual error message (e.g. "404 Not Found") rather than a
generic "Could not load reports."
- Detects 404 specifically and prompts: "The /api/briefs/:id/reports endpoint
is missing on the server. Pull the latest main and run
`docker compose ... up -d --build`."
Together these answer the "is the server actually running new code?" question
without needing to SSH and grep docker logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The inline DEMO_BRIEF in list.tsx was beefed up in 15aa5a6, but the file at
v2/examples/dove-demo-brief.json that ships in the repo wasn't — it still had
the old thin shape (3 competitors, 2 KPIs, 4 interests, one-line positioning).
Mirroring the inline content so anyone downloading the example file gets the
same realistic example as the "Load Dove demo" button.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Dove demo was minimal — 3 competitors, 2 KPIs, 4 interests, one-line
positioning. New users importing it would see a thin brief and assume the
app expected thin briefs. Beefed up to actually demo the brief shape:
- 7 competitors (Olay, Garnier, Pantene, Nivea, Cerave, Aveeno, Tresemme)
- 5 KPIs reflecting how a brand strategist actually frames trend reports
(cultural territory, hook patterns, emerging behaviours, paid/organic
map, sentiment risk)
- 8 audience interests including the everything-shower / anti-influencer
vocabulary the brief is meant to surface
- Proper "real beauty, real care" positioning paragraph
- Detailed business question with both why-now and what-should-Dove-do
- Substantive context_vision explaining what success looks like for the
demo run vs the follow-on $200 brief with MoM compare
- $50 budget (was $30) so 5+ scrapes can run before the Pass-1 cap
- Explicit prior_report_id: null so the field is visible in the editor
Both the operator-app inline DEMO_BRIEF and v2/examples/dove-demo-brief.json
updated to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After importing a brief via JSON paste, edit appeared not to work — the
textarea would either be empty or silently revert as React Query refetched
on window focus (every refetch fired the prefill useEffect, blowing away
whatever the user had typed). On top of that, if the server's `full` field
wasn't returned for any reason the textarea stayed permanently empty.
Fixes:
- Initialise the textarea exactly once via a useRef seed flag. Subsequent
data refetches don't overwrite user-typed content.
- When `full` is missing, fall back to reconstructing the BRIEF_INPUT shape
from public columns (client_name, brand positioning, kpis, quality floor,
etc.) so the user has something to edit. Surfaces an amber banner noting
competitors/audience/geo are blank and need re-entering.
- New "Reset to saved" link in the header to deliberately discard local
changes and reload from the server.
- Disable Save when the textarea is empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Dove demo brief failed at Stage 3 with "Spent $0.00 across 0 scrapes"
— Stage 2 didn't actually invoke Apify even once. Two failure modes were
silent:
1. APIFY_LIVE_APPROVED is not 'true' in the container env, so every
runActor() call returns {items: [], status: 'DRY_RUN'} with no spend
log entry. Stage 2 happily writes empty pass1_videos.json and writes
its sentinel; Stage 3 then can't tell why.
2. seeds.json has zero hashtags + zero handles + zero search_terms (a
degenerate Stage 1 result), so the Stage 2 priority loop has nothing
to iterate.
Now Stage 2 throws fast on both, before scheduling any work:
- "Apify is in DRY-RUN mode but brief budget > 0. Set APIFY_LIVE_APPROVED=true
in v2/.env, restart V2, click Force re-run."
- "seeds.json contains zero hashtags/handles/search_terms. Click Force re-run
to regenerate Stage 1."
Both messages flow through to the Reports detail page failure panel via the
existing error_message COALESCE plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 1000 likes / 10000 plays defaults are calibrated for top-of-funnel
beauty/fitness scrapes; in narrower TikTok niches almost every video lands
below them and Stage 2 returns 0 keepers. Defaults dropped to 100 likes /
1000 plays across:
- server/schemas/brief.ts (Zod default)
- db/init.sql (column default for new DBs)
- examples/dove-demo-brief.json
- operator-app's brief-form initial values
- operator-app's "Load Dove demo" inline brief
Stage 3 empty-pass1 error now reads pass1/spend_log.json and reports the
actual scrape breakdown — total $ spent, total raw videos returned, and
how many got dropped by each floor (zero-plays / min_plays / min_likes /
min_stl_pct). So instead of a generic "lower the floor", the user sees:
"Spent $5.42 across 7 scrapes; 1400 videos returned. Dropped: 12 zero-plays,
1305 below min_plays=10000, 31 below min_likes=1000."
Existing briefs are unaffected (column default applies to NEW rows). For the
in-flight Dove2 run the user can edit the brief and lower the floor, then
click Retry pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When you navigated away from a brief and came back, there was no indication
that a pipeline had already run for it — the page just showed the brief
fields and a "Run pipeline" button, making completed/in-flight runs invisible
without first hitting Home.
Now the brief detail page renders a "Reports for this brief" section listing
every run for the brief — status pill, run id, total cost, started/finished
relative timestamps, click-through to the run page. Auto-refreshes every 3s
while any run is non-terminal so an in-flight pipeline shows live progress
even when the user navigated to the brief instead of the report page.
Server:
- db/reports.ts: listReportsForBrief(brief_id, limit).
- routes/reports.ts: handleListReportsForBrief.
- index.ts: GET /api/briefs/:id/reports.
Client:
- api/reports.ts: useReportsForBrief hook with conditional polling.
- routes/briefs/detail.tsx: BriefReports section with status pills, in-flight
shortcut link, empty state when no runs exist yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues addressed:
1) Delete brief was awkward — only on the detail page, only for admin/owner,
and the on-disk briefs/<id>/ tree was orphaned even after a successful
DB delete. Server policy is now:
- team owner / admin: delete any brief in team
- team editor: delete only briefs they own
- team viewer: cannot delete
- super-admin: bypass
Plus: refuses to delete if a non-terminal pipeline run is active for the
brief, and rm -rf's briefs/<id>/ after the FK cascade so scrape data,
datasets and dashboards don't pile up.
Operator UI: per-row Delete button on the briefs list page (with confirm)
for users who have delete rights on each brief; detail-page rule loosened
to match the server's editor-owner policy.
2) Pipeline error messages were being clobbered. The server's child-on-exit
handler called finishReport() unconditionally, overwriting the actual
error the pipeline's main()-catch had just written. Now we use a
COALESCE update that only sets error_message when it's still NULL —
so the real failure (e.g. "Stage 3: pass1_videos.json is empty…") wins
over the generic "Pipeline exited with code 1". Stage 3's empty-pass1
error message is also now diagnostic: it lists the brief's current
engagement-floor values and suggests what to lower.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pipeline `all` command now skips stages whose .state/stage{N}.done
sentinel is present (unless --force), so retrying picks up exactly where
the previous run failed without re-spending on completed stages. Stage 5
(validate) is the deliberate exception — it always re-runs because
--drop-failing changes the selection set.
- pipeline/cli.ts: `all` wraps every stage in a maybeRun() helper that
checks the sentinel + writes one after running. Validate runs every
retry; if it doesn't reach 100% coverage the pipeline now fails LOUDLY
with a clear error rather than crashing in Stage 6.
- routes/reports.ts: handleRetryReport — POST /api/reports/:id/retry,
resets reports.status to pending (clears error_message + finished_at),
spawns `pipeline cli all` with --drop-failing (and --force when the
client passes {force:true}). Same singleton-running guard as run.
- operator-app: useRetryReport hook + two new buttons on Reports detail:
- "Retry pipeline" / "Force re-run" on the failure panel.
- "Retry with drop-failing" on the manifest panel (Stage 5 specific).
Items explicitly deferred (documented in head):
- SSE for live progress (3s React Query polling already in place).
- Conversational CLI brief intake (V3 §0; the operator-app form covers it).
62/62 unit tests pass. SPA build 284 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The V2 container runs as uid 1000 (Dockerfile.v2:39 USER 1000). When the
pipeline tried to mkdir briefs/<brief-id>/ on the production server, it hit
EACCES because the host directory was owned by root (V1 had a similar fix
in deploy.sh that we never ported).
- cutover-in-place.sh: chown -R 1000:1000 briefs/ before docker up.
- deploy-v2.sh: same chown on every redeploy + use --env-file (was missing).
Immediate manual fix on the running server (until next deploy):
sudo chown -R 1000:1000 /opt/social-reporting/briefs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the last big spec gap: a single bundled React+Vite+Tailwind+Recharts
SPA that renders any report's dataset_v2.json under /api/reports/:id/dashboard/.
Spec deviation (intentional): rather than per-brief Vite builds (Netlify-
portable case), one bundle serves every report and fetches its dataset at
runtime. The portable HTML bundle (§10b) still ships per-report for offline
upload to claude.ai.
Dashboard template (v2/templates/dashboard_template):
- React 18 + Vite + TypeScript + Tailwind + Recharts. Same dark theme tokens
as the operator app.
- 9 views: Overview, Categories, Trends explorer (filters + sort + drilldown),
Lenses (4 sub-tabs for Hooks Library / Visual Vernacular / Audio Atlas /
Sentiment Map), Charts (engagement-vs-reach scatter, category treemap,
paid-vs-organic stacked bar), Compare (MoM new/returning/faded + category
momentum), Paid creators appendix, Methodology.
- Tab-based navigation with hash-stable URLs so deep links work.
Server:
- GET /api/reports/:id/dataset returns the on-disk dataset_v2.json (auth-
gated; 404 if Stage 10 hasn't built it yet).
- handleDashboardServe rewrite: serves (1) the explicit dashboard.html bundle,
(2) per-report on-disk static (covers, dataset_v2.json), (3) the bundled
SPA's assets/* and index.html with SPA fallback.
Build:
- v2/package.json workspaces re-includes templates/dashboard_template (was
excluded after earlier cutover-prep).
- Dockerfile.v2 builds the dashboard template alongside operator-app and
copies dist/ into the runtime image.
Local build: 838 modules transformed, 580 kB / 166 kB gzipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the V3 brief gaps that were called out in the audit.
Stage 8c lens artefacts (V3 §8c):
- prompts/lens_enrichment.md: rubric for Hooks Library / Visual Vernacular /
Audio Atlas / Sentiment Map.
- stages/stage_8c_lenses.ts: zod-validated lens generation from atomic
insights + per-video analyses; writes lenses/{hooks_library,visual_vernacular,
audio_atlas,sentiment_map}.json.
- stage_10_build.ts: dataset_v2.json now includes the four lens arrays.
- cli.ts: new `lenses` command; `all` runs 8c after Stage 8 (fail-soft);
`build` runs 8c too in case the previous run skipped it.
Pipeline split for sign-off enforcement (V3 §9):
- cli.ts `all` command now stops at status=qa after Stage 9. The operator app
drives Stage 10 separately via POST /api/reports/:id/build, which the
server only allows after CM + Strategist sign-offs from two different
humans.
- routes/reports.ts: handleQaSignoff (POST /api/reports/:id/qa/sign with
role=cm|strategist), handleBuildReport (verifies both signoffs + different
user_ids, then spawns `cli.ts build`). handleGetReport now also returns
manifest summary + qa.{cm_signoff,strategist_signoff}.
Brief edit (PATCH):
- db/briefs.ts: updateBrief.
- routes/briefs.ts: handleUpdateBrief, with editor+ team role.
- /api/briefs/:id PATCH route added.
- operator-app: useUpdateBrief hook; new /briefs/:id/edit route — minimal
JSON-textarea form, prefilled from brief.full, with Zod-issue surfacing.
- briefs/detail: "Edit" button next to Export/Run.
Reports detail UI:
- ManifestPanel: when manifest summary is in the response, render asset-
status grid + collapsible missing-videos list + the exact CLI command to
--drop-failing-backfill.
- SignoffPanel: two cards (CM + Strategist) showing signed-by-email/at;
"Sign as ..." button per side; client-side guard prevents the same user
signing both; "Build report" button enabled only when both signoffs
present + different humans.
- Dashboard static-serve route + Open dashboard / Download bundle from
earlier session re-confirmed wired.
Server clean, vite build green at 282 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- briefs/list: new "Import JSON" button; collapsible panel with file upload,
paste textarea, "Load Dove demo" button (inlines the same demo we ship
in v2/examples/dove-demo-brief.json). On submit POSTs the JSON to
/api/briefs and surfaces server-side Zod issues inline.
- briefs/detail: new "Export JSON" button — downloads `<slug>.brief.json`
using the brief_yaml the server now exposes under `full`.
- v2/examples/dove-demo-brief.json: real Dove TikTok demo brief, $30
budget, ready to run end-to-end via the Run pipeline button.
- pipeline/cli.ts: usage() text de-stale-ified — every stage is real now,
the "TODO Phase X" tags removed; new commands documented (`all`,
`backfill-covers`, `--run-id`, `--drop-failing`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
enable mp4 download, smart Stage 4 cache, default Apify live in prod
Closes the last gaps so the operator app is end-to-end usable in production.
Server:
- routes/reports.ts: GET /api/reports/:id/dashboard[/<file>] serves files
out of the report's brief outputs/ tree (HTML bundle, dataset_v2.json,
any covers referenced relatively). Auth-gated by team viewer role.
Path-traversal guarded.
- index.ts: two new route patterns (with and without trailing path).
Client:
- routes/reports/detail.tsx: "Open dashboard" is a target=_blank anchor at
/api/reports/:id/dashboard/, "Download" same URL with download attribute.
No more dead SPA-internal link.
Pipeline polish (the four open items from the smoke test):
- stage_10_build.ts: covers are now downscaled via ffmpeg (240px / q=6)
before base64 inlining. Hard ceiling per cover 60 KB; falls back to the
original only if it already fits. Honours V3 brief's ≤3 MB HTML bundle.
- lib/apify_client.ts: post-run cost is re-polled with backoff (0/5/15/30s)
instead of a single read. TIKTOK_COMMENTS reports $0 immediately and
$5+ later — without this the soft cap can't fire on it.
- stage_2_pass1_scrape.ts: shouldDownloadVideos:true (and shouldDownloadCovers:true)
by default so videoMeta.downloadAddr is populated for Stage 4 frame
extraction. Disable with DISABLE_VIDEO_DOWNLOADS=true if the budget is
tight.
- stage_4_pass2_enrich.ts: Stage 5 backfill candidates aren't in the
transcripts/comments cache. New loadOrFetchActor() reads what's cached,
identifies missing ids, fetches just those from Apify, and merges back
into the cache. Backfill no longer drops every candidate.
Production defaults:
- .env.example: APIFY_LIVE_APPROVED=true (commented; operators can flip
to false for dry-runs).
- cutover-in-place.sh: sets APIFY_LIVE_APPROVED=true if not already in .env
after the migration step, so a fresh prod cutover doesn't accidentally
dry-run.
62/62 unit tests pass; tsc + vite build green; bundle 269 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the gap between "brief exists" and "report ships". The Phase A
placeholders for Home and Reports/detail are now real, and the brief detail
page can actually start a pipeline run.
Server (no schema changes — reports table already existed):
- db/reports.ts: createReport, getReport, getReportWithBrief, listReportsForTeam,
updateReportStatus, finishReport, logCostEvent (atomically updates the
reports row's running totals), listCostEvents.
- routes/reports.ts: GET /api/reports (active team), GET /api/reports/:id
(with cost_events), POST /api/briefs/:id/run that
1. authorises (editor+ on the brief's team),
2. creates a reports row (status=pending),
3. spawns the pipeline as a detached child running
`tsx pipeline/cli.ts all --report <brief_id> --run-id <reports.id>`,
4. returns the new report id.
Singleton flag prevents two concurrent runs (mirrors V1).
Pipeline:
- cli.ts: new --run-id flag. New `all` command drives every stage in order
via a withStage() helper that updates reports.status / current_stage at
each step. Cost callbacks now ALSO write to cost_events when run-id is
set, tagged with the current stage. main()'s catch handler calls
finishReport(runId, 'failed', err.message) so the UI doesn't poll forever
on a crash.
Client:
- api/reports.ts: useRecentReports, useReport (auto-polls every 3s while
status is non-terminal), useRunPipeline.
- routes/home.tsx: real recent-reports list — status pill, brief client +
business question, cost split, relative time.
- routes/reports/detail.tsx: full run page — header with status pill,
10-step pipeline progress with current-stage pulse, error block on
failure, three-tile cost summary (total / apify / claude), cost-event
log (most recent first, scrollable, sticky header), "Open dashboard"
+ "Download HTML bundle" actions when the run completes.
- routes/briefs/detail.tsx: Run pipeline button is now functional for
editors+, with a confirm dialog (warns about Apify/Claude spend),
navigates to the new /reports/:id on success, surfaces 409 if another
run is in flight.
62/62 unit tests still pass. Typecheck + vite build green; bundle 269 kB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the three Phase A scaffold placeholders with working pages backed by
the existing server endpoints (no server changes).
- src/api/teams.ts (new): useTeams, useTeam, useCreateTeam, useAddMember,
useUpdateMemberRole, useRemoveMember. All with cache invalidation.
- src/api/admin.ts (new): useAllUsers, useToggleSuperAdmin.
- routes/teams/list.tsx: list of teams (cards with role badge + slug + Personal
marker), inline "Create a team" form (POSTs /api/teams), each card links to
/teams/:id. Inline 409 / validation handling.
- routes/teams/detail.tsx: team header with my role; members table; owners can
change member roles via dropdown, owners + admins can remove members
(confirm dialog); below the table, an Invite form (email + role select)
matching POST /api/teams/:id/members. Per-row + per-form error surfacing.
- routes/admin/users.tsx: full users table — email, name, super-admin badge,
created/last-login timestamps, promote/revoke button. Disables the toggle
for the current user when they're super-admin (matches the server's
self-demotion guard); also shows "(you)" indicator next to your own row.
Bundle size: 258 kB → /social-reports/assets/index-C0ofQc9Y.js (+7 kB gzipped).
TS strict + vite build pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The server returns { user, teams, active_team } (server/routes/me.ts:19-30).
The SPA's MeResponse type had `memberships`, so TeamSwitcher's `data.memberships.length`
crashed on initial render after sign-in:
TypeError: Cannot read properties of undefined (reading 'length')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues from the first server cutover:
1. SPA loaded white-on-black blank because import.meta.env.VITE_AZURE_TENANT_ID
and VITE_AZURE_CLIENT_ID were undefined at runtime. Vite reads VITE_* at
*build time* and inlines them into the bundle; passing them only as
runtime container env vars is too late.
- Dockerfile.v2: declare ARG VITE_AZURE_TENANT_ID, VITE_AZURE_CLIENT_ID,
VITE_BASE; export as ENV before `npm run build`.
- docker-compose.v2.yml: forward AZURE_TENANT_ID / AZURE_CLIENT_ID /
VITE_BASE through `build.args` so the cutover .env values reach Vite.
2. cutover-in-place.sh stopped V1 with `-p social-listening`, but V1's actual
compose project name was `social-reporting` (parent dir). Old V1 containers
were left running. Now we try both project names AND sweep by container
name pattern (anything matching social-listening or social-reporting-db-1
that isn't a V2 container).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The host port 5437 on the optical-dev server was already allocated by
something (probably an old stopped-but-not-removed Postgres container
or another tracking app). V2 doesn't need a host port for db-v2 in
production — app-v2 reaches it over the docker network at db-v2:5432.
Per CLAUDE.md "always check for ports that are already used":
- docker-compose.v2.yml: remove the unconditional db-v2 host port
binding. Compose's list-merge semantics meant `ports: []` in the prod
override didn't actually clear the base list.
- docker-compose.v2.dev.yml (new): local-dev overlay that re-adds the
host port for psql convenience. Use with `-f base -f dev`. Bound to
127.0.0.1 so the db is never reachable from outside the dev machine.
- cutover-in-place.sh: pre-flight check on APP_V2_PORT (3457) — if
it's held by something other than our own V2 container, abort with
a clear message rather than failing mid-deploy.
Verified locally: `compose -f base -f prod up -d` brings up a stack
with db-v2 having no host port (just internal 5432/tcp), app-v2 on
127.0.0.1:3457, /api/health returns {ok:true}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The unanchored `briefs/` rule (intended for the runtime per-report tree at
the repo root) also matched `v2/operator-app/src/routes/briefs/` — so the
brief-form components never made it into git, and the prod docker build
failed at vite with "Could not resolve ./routes/briefs/list".
Fix: anchor to the two specific paths that should be ignored, /briefs/
and /v2/briefs/. The React routes dir under src/ is now tracked.
Adds the four missing files: list.tsx, new.tsx, detail.tsx, _form-bits.tsx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A scaffolded the SPA at the bare origin (`/`); production lives behind
Apache at `/social-reports/`. Without these fixes, V2's built assets 404 and
Azure SSO rejects the redirect URI mismatch.
- Vite `base: /social-reports/` (overridable via VITE_BASE for dev).
- BrowserRouter basename = import.meta.env.BASE_URL.
- apiFetch + msal-browser script src + token-exchange URL all prefix BASE.
- MSAL redirectUri now matches V1's Azure-registered URI:
`${origin}/social-reports/login.html`.
- New `<Route path="/login.html">` alias renders the same Login component
so React Router matches the redirect URI when MSAL returns.
Deploy ergonomics (the user wants V1 gone from the server):
- v2/deploy/cutover-in-place.sh: run from /opt/social-reporting; stops V1,
pulls main (v2/ appears, V1 dirs deleted), migrates secrets from V1's
.env into v2/.env, swaps Apache, starts V2. Single command, no clone of
a sibling dir needed.
- setup-v2.sh: PURGE_V1=true flag now cleans /opt/social-reporting and
the V1 docker volume after V2 is healthy.
- rollback-to-v1.sh: re-clones the v1-archive branch when V1 is no longer
on disk (REPO_URL required).
62/62 unit tests still pass; vite build emits assets under /social-reports/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V1's running deployment at /opt/social-reporting on the server stays put
until cutover; V1's source is preserved on the v1-archive branch and via
git history. From this commit forward, all work targets v2/.
The new root README points contributors at v2/ and documents the rollback
path (deploy/rollback-to-v1.sh) for the cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V2 lives entirely under v2/ and is built around three asks the team raised
about V1: per-video assets sometimes drifted onto the wrong trend, hashtag
scrapes returned junk that wasn't filterable per-client, and there was no
multi-user model behind Microsoft SSO.
Highlights:
- Stable TikTok numeric-id key for every per-video asset; URL form drift is
logged loudly to drift_log.jsonl and never silently nulls assets. Stage 5
manifest hard-gates Stage 6 if any selected video is missing any required
asset; --drop-failing auto-backfills from the next-best recipe candidates.
- Per-brief engagement floor (min_likes / min_plays / min_stl_pct), applied
at Apify scrape time and re-validated locally; spend_log.json records
raw_returned vs kept_after_floor per scrape.
- Users + teams + memberships with owner/admin/editor/viewer roles; SSO
upserts a user keyed on Azure oid, auto-creates a personal team, and a
super-admin is bootstrapped via BOOTSTRAP_SUPER_ADMIN_EMAIL on first
sign-in. Phase A integration test: 16/16 pass.
- 10-stage TS pipeline (brief → seed → scrape1 → select → scrape2 →
validate → analyse → insights → trends → qa → build) wired through one
CLI; each stage idempotent + resumable from disk via .state sentinels.
§4.5 rubrics shipped under prompts/ and loaded into Claude calls.
- React 18 + Vite + TS + Tailwind operator SPA: brief intake form,
team management, super-admin user list, help/FAQ ported from V1.
- Separate Docker Compose project (name: social-reporting-v2, port 3457,
Postgres 5437) with deploy/setup-v2.sh, deploy-v2.sh, rollback-to-v1.sh
scripts that take over V1's /social-reports URL and let us roll back.
Verification: 62 unit tests pass (auth/session, ids extractor with full URL
fixture, engagement floor, recipes, manifest, linking-fix, MoM compare).
Live smoke run on a Dove brief: 1400 raw → 253 kept (82% culled) → 21
fully-bundled videos → 25 editorial trends across 8 brief-driven categories,
with drift=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace cp frontend/* with cp -r frontend/. to copy all files reliably
- Add mkdir -p as safety net in deploy.sh
- Add apache2 reload after frontend copy in deploy.sh
- setup.sh now copies entire frontend dir instead of hardcoded filenames
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Optional textarea lets users provide strategic guidance like objectives,
competitive context, and focus areas. Injected into Claude prompts at
stages 2, 4, 6, and 8 so all agents can produce more focused output.
Backward compatible — empty context changes nothing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add apifyBudget field to ClientBrief (default $10)
- Budget split: 70% discovery (evenly across platforms), 30% enrichment
- Per-platform soft cap prevents one platform hogging the budget
- Budget input field added to both frontend and dashboard forms
- Saved briefs preserve budget setting
- Fix Claude Vision 5MB limit: filter oversized thumbnails before batching
- Fix Docker: ensure node user can write to volume-mounted dirs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix TikTok comments actor input: `videos` → `videoUrls` (wrong field name)
- Fix TikTok transcripts actor input: `videos` → `videoUrls` (wrong field name)
- Allow HTTP URLs for thumbnails (TikTok CDN uses HTTP)
- Add date filtering to profile scrapers (TikTok + Instagram)
- Keep videos with unparseable dates instead of dropping them
- Lower visual language threshold from 5 to 3 thumbnails
- Increase thumbnail timeout from 5s to 10s
- Add logging for failed thumbnail downloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Remove Desk Research (Stage 7 skipped, sources removed from report)
2. Fix comments scraping: increase cap to 2000, handle alt field names
3. Dynamic stats bar: hide zero-value stats instead of showing "0 Comments"
4. Prompt improvements: enforce timeliness, comment-based insights, creator spotlight algorithm (2-10 videos, exclude >50% dominance)
5. Date filtering: pass date params to Apify actors (oldestCreateTime, onlyPostsNewerThan, uploadDate) + log filter counts
6. Pullquotes: 3-4 generated editorial dividers between sections
7. Thumbnails: download top 50 coverUrl as base64, store on EnrichedVideo
8. Visual Language section: 5 batches of 10 through Claude Vision, synthesized into 5-6 visual codes with thumbnail cards
9. Sticky navigation bar with anchor links to all sections
10. New types: VisualCode, thumbnailUrl on Video, thumbnailBase64 on EnrichedVideo, pullquotes/visualCodes on ReportJSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
