fb5acdb892
4-stage agent pipeline (Data Mapper, Input Collector, Strategy Writer, Report Builder) with React wizard, Postgres persistence, HITL + YOLO modes, Apify embed hydration, clone-for-next-month, and slide-deck HTML output. Proven end-to-end against real Cif Meltwater data (Instagram + TikTok) with Anthropic Opus 4.7 (strategy) and Sonnet 4.6 (report builder). |
||
|---|---|---|
| backend | ||
| Build-Information | ||
| frontend | ||
| infra | ||
| scripts | ||
| .env.example | ||
| .gitignore | ||
| CLAUDE.md | ||
| docker-compose.override.yml | ||
| docker-compose.yml | ||
| README.md | ||
| STATUS.md | ||
Social MI/BI — Multi-Agent Reporting
A web app that reproduces and extends the agency's four-agent social-media reporting workflow (Data Mapper → Input Collector → Strategy Writer → Report Builder). It turns monthly Meltwater / Meta exports into a standalone slide-deck HTML report — with a React wizard, Postgres-backed persistence, HITL checkpoints, optional YOLO auto-advance, Apify embed hydration, and clone-for-next-month.
Architecture
┌─────────────────────────────────────────────────────┐
│ React + Vite SPA │
│ │
│ Login → Dashboard → Report Wizard (stepper) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │
│ │Data Mppr │ │Input Col │ │Strategy │ │Report │ │
│ │Panel │→│Panel │→│Panel │→│Panel │ │
│ └──────────┘ └──────────┘ └──────────┘ └────────┘ │
│ │ │ │ │ │
│ └── CheckpointPanel (approve / edit / reject)│
└────┬──────────────┬─────────────────────┬───────────┘
│ REST /api │ WebSocket /ws │
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ FastAPI (uvicorn) │
│ │
│ routes_auth routes_brands routes_reports │
│ routes_stages routes_uploads routes_embeds │
│ ws_stage_events (Redis pub/sub fan-out) │
└────┬──────────────┬──────────────┬──────────────────┘
│ arq enqueue │ SQLAlchemy │ boto3
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Redis │ │ Postgres │ │ MinIO │
│ (queue + │ │ (state + │ │ (uploads │
│ pub/sub)│ │ artefacts│ │ +reports)│
└────┬─────┘ └──────────┘ └──────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ arq Worker │
│ │
│ StateMachine.run_stage(report_id, stage_no) │
│ │ │
│ ├── Stage 1: Data Mapper (deterministic; │
│ │ openpyxl → Meltwater IG + TikTok adapters)│
│ │ │
│ ├── Stage 2: Input Collector (deterministic; │
│ │ validates logos / embeds / insights) │
│ │ │
│ ├── Stage 3: Strategy Writer (Anthropic │
│ │ Opus 4.7, streaming, prompt caching) │
│ │ │
│ └── Stage 4: Report Builder (Anthropic │
│ Sonnet 4.6, streaming, emits slide deck) │
│ │
│ After each stage: │
│ • stage_artefacts row written (JSON or HTML) │
│ • top_posts / learnings projected for UI │
│ • token usage + cost_cents recorded │
│ • WS events published (stage_status / │
│ awaiting_review / stage_complete / tokens) │
│ • HITL mode → pause at awaiting_review │
│ • YOLO mode → auto-advance to next stage │
└─────────────────────────────────────────────────────┘
External services: Phase-2 add-ons:
• Anthropic (Claude) • Apify (TikTok + IG scrapers)
• Google Fonts (Inter) • Playwright (HTML → PPTX)
• Chart.js CDN • MSFT SSO (Phase 3)
The wizard is the control surface; the worker does the work; every artefact is persisted so you can pause, edit, resume, or clone any report at any stage. The WebSocket stream is eventually-consistent sugar on top — the source of truth is always Postgres.
Quick start (dev)
Prereqs: Docker Desktop, ~4 GB free, no services already on 5173 / 8088 / 5433 / 6380 / 9002 / 9003.
# One-time
cp .env.example .env
openssl rand -hex 32 # paste into SESSION_SECRET in .env
# Optional but required for stages 3–4: set ANTHROPIC_API_KEY
# Optional for phase-2 embed hydration: set APIFY_API_TOKEN
# Bring up storage + seed schema
docker compose up -d db redis minio minio-init
docker compose run --rm api alembic upgrade head
# Start the app
docker compose up -d api worker web
Open:
- Web UI — http://localhost:5173
- API docs — http://localhost:8088/docs
- MinIO console — http://localhost:9003
Log in with BOOTSTRAP_ADMIN_EMAIL / BOOTSTRAP_ADMIN_PASSWORD from .env
(the first admin is created on first boot only).
End-to-end smoke against real Cif data
# From inside the api container
docker compose exec -e PYTHONPATH=/app api python scripts/run_e2e_cif.py
Writes rendered/cif-e2e-{report_id}.html — a ~55 KB standalone slide deck.
Ports
Defaults; override in .env if conflicted.
| Service | Host port |
|---|---|
| API | 8088 |
| Web (Vite) | 5173 |
| Postgres | 5433 |
| Redis | 6380 |
| MinIO S3 | 9002 |
| MinIO Console | 9003 |
Repository layout
SOCIAL-MI-BI/
├── docker-compose.yml ─ dev stack (db / redis / minio / api / worker / web)
├── docker-compose.override.yml ─ dev-only hot-reload + volume mounts
├── infra/ ─ Dockerfiles, nginx conf, minio init
├── Build-Information/ ─ spec inputs (READ-ONLY)
│ • Agents/Agent 1..4.txt ─ canonical agent prompts (verbatim)
│ • Workflow/*.html ─ kick-off workflow description
│ • Sample Data Input/*.xlsx ─ real Cif Meltwater exports (fixtures)
├── backend/
│ ├── pyproject.toml
│ ├── alembic/ ─ schema migrations
│ └── app/
│ ├── main.py ─ FastAPI factory + lifespan bootstrap
│ ├── config.py ─ pydantic-settings (env-driven)
│ ├── db.py ─ async SQLAlchemy session scope
│ ├── models/ ─ ORM (Brand, ReportRun, StageExecution,
│ │ StageArtefact, TopPost, Learning, …)
│ ├── schemas/ ─ pydantic DTOs (auth, brand, report)
│ ├── api/ ─ routers_* + ws_stage_events
│ ├── services/
│ │ • storage.py ─ MinIO / S3 wrapper (dual endpoint)
│ │ • anthropic_client.py ─ Claude SDK; always-stream, cache-aware
│ │ • cost_tracker.py ─ cents-per-MTok rate table
│ │ • apify_client.py ─ TikTok + IG actor calls, 30d cache
│ │ • pubsub.py ─ Redis pub/sub for WS events
│ │ • auth.py ─ bcrypt + itsdangerous sessions
│ │ • bootstrap.py ─ seed platforms, benchmarks, admin, buckets
│ ├── agents/
│ │ • prompts/ ─ VERBATIM copies of Agents/Agent 1..4.txt
│ │ • base.py ─ StageBase / StageInput / StageOutput
│ │ • data_mapper.py ─ (deterministic) runs ingest adapters
│ │ • input_collector.py ─ (deterministic) validates embeds/insights
│ │ • strategy_writer.py ─ Opus 4.7, British-English guard
│ │ • report_builder.py ─ Sonnet 4.6, deck-skeleton-guided HTML
│ ├── ingest/ ─ openpyxl Meltwater IG / TikTok adapters
│ ├── workflow/
│ │ • state_machine.py ─ per-report stage lifecycle
│ │ • queue.py ─ arq WorkerSettings
│ │ • events.py ─ stage event names / publisher
│ ├── report/
│ │ • deck_skeleton.html ─ style reference (slide deck)
│ │ • pptx_converter.py ─ phase-2 HTML→PPTX via Playwright
│ └── tests/ ─ unit / integration / e2e (real Cif data)
├── frontend/
│ ├── vite.config.ts ─ proxy /api → api container
│ └── src/
│ ├── pages/ ─ Dashboard, NewReport, ReportWizard, …
│ ├── components/
│ │ • wizard/ ─ Stage*Panel + CheckpointPanel
│ │ • EventLog.tsx ─ coalesced token stream viewer
│ │ • SlidePreview.tsx ─ iframe over the export endpoint
│ ├── lib/ ─ api.ts, ws.ts, auth.ts, toast.ts
│ └── styles/globals.css ─ Tailwind + amber (#FFC407) palette
├── scripts/
│ ├── run_e2e_cif.py ─ one-shot end-to-end harness
│ └── seed_benchmarks.py ─ idempotent benchmark seed
├── rendered/ ─ generated decks land here (git-ignored)
├── STATUS.md ─ current delivery status / verification log
└── README.md ─ this file
Stage pipeline
| Stage | Agent | Model | Streaming | Deterministic? |
|---|---|---|---|---|
| 1 | Data Mapper | — | — | Yes (Python + openpyxl) |
| 2 | Input Collector | — | — | Yes (validation heuristics) |
| 3 | Strategy Writer | claude-opus-4-7 |
yes | No — LLM narrative |
| 4 | Report Builder | claude-sonnet-4-6 |
yes | No — LLM HTML slide deck |
Stage 4 receives the deck skeleton (backend/app/report/deck_skeleton.html,
modelled on the Monks.Flow reference deck) as a cache-controlled system
block, plus the brand, data-mapper JSON, input-collector JSON, and
strategy-writer learnings. The emitted HTML always contains a Chart.js CDN
tag, .slide.active transitions, a keyboard handler for arrow-key
navigation, and --brand-primary / --brand-secondary / --accent CSS
variables seeded from the brand row.
HITL mode pauses after every stage at status awaiting_review. The Check-
pointPanel exposes approve / reject-with-feedback / edit-inline / re-run.
YOLO mode auto-advances end-to-end but still streams events.
Resume-from-stage: POST /reports/{id}/stages/{n}/run on a completed stage
re-enqueues stage n; stages n+1..4 cascade because the state machine
rebuilds StageInput from the latest artefacts.
Clone-for-next-month: POST /reports/{id}/clone copies brand, up-front
context, platform scope; sets reporting_month to parent + 1 month; fresh
uploads are then required.
Data retention / GDPR
DELETE /api/v1/reports/{id} (admin-only) cascades every row tied to the
report (stage executions, artefacts, top posts, embeds, insights, learnings,
uploaded files) and purges the corresponding MinIO keys. audit_events rows
are kept but their payloads are scrubbed.
Verification
docker compose exec -T api pytest tests/unit tests/integration -v
Runs:
tests/unit/test_ingest.py— the two Meltwater adapters against the real Cif Instagram and TikTok Excels. Assertsposts_current = 95 / 93,posts_prev = 76 / 77, engagement-rate deltas, top-3 URLs by platform.tests/unit/test_data_mapper_agent.py— full Stage 1 StageOutput against the Agent 1 JSON contract.tests/integration/test_api_smoke.py—/healthz+ login +/auth/me.
tests/e2e/test_cif_pipeline.py runs all four stages (skipped unless
RUN_LIVE_E2E=1 and ANTHROPIC_API_KEY are set).
Phase status
- Phase 1 — MVP (done). HITL wizard, Postgres persistence, 4 agents proven end-to-end against real Cif data, HTML slide-deck output, brand-theming via CSS variables, admin/user auth.
- Phase 2 — in flight. YOLO mode live in the state machine; Apify
client wired (manual-paste fallback today); PPTX converter scaffolded
behind
pip install .[phase2]+playwright install chromium; clone and resume-from-stage working; MSFT SSO deferred. - Phase 3 — planned. MSFT SSO (Entra ID / MSAL), auto-brand discovery (search + logo scrape + dominant-colour extraction), cost dashboard.