Commit graph

17 commits

Author SHA1 Message Date
DJP
f2d6f56831 Report quality overhaul: 11 feedback items
1. Remove Desk Research (Stage 7 skipped, sources removed from report)
2. Fix comments scraping: increase cap to 2000, handle alt field names
3. Dynamic stats bar: hide zero-value stats instead of showing "0 Comments"
4. Prompt improvements: enforce timeliness, comment-based insights, creator spotlight algorithm (2-10 videos, exclude >50% dominance)
5. Date filtering: pass date params to Apify actors (oldestCreateTime, onlyPostsNewerThan, uploadDate) + log filter counts
6. Pullquotes: 3-4 generated editorial dividers between sections
7. Thumbnails: download top 50 coverUrl as base64, store on EnrichedVideo
8. Visual Language section: 5 batches of 10 through Claude Vision, synthesized into 5-6 visual codes with thumbnail cards
9. Sticky navigation bar with anchor links to all sections
10. New types: VisualCode, thumbnailUrl on Video, thumbnailBase64 on EnrichedVideo, pullquotes/visualCodes on ReportJSON

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 09:52:08 -04:00
DJP
3dcdf0cc69 Add project README with architecture, setup, and deployment docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 14:13:24 -04:00
DJP
2429deff72 Round cost displays to nearest cent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 13:43:41 -04:00
DJP
4e16367d2d Fix brief loading: remove stale jsonPreview refs, add Export button, rename Load
- Fixed null reference error when loading JSON files (removed deleted jsonPreview element refs)
- Added Export button to download saved briefs as JSON files
- Renamed "Load & Run" to just "Load" per user feedback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 13:34:34 -04:00
DJP
010d304c2a Add saved briefs feature: server-side storage with dedicated tab
- Backend: GET/POST/DELETE /api/briefs endpoints storing JSON files in briefs/ dir
- Frontend: new Saved Briefs tab with cards showing client details, Load & Run, Delete
- Save Current Brief button on Pipeline tab persists form to server
- Both standalone dashboard and static frontend updated

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 13:30:15 -04:00
DJP
5f8d84f5c5 Add delete runs, bulk clear, and report download to dashboard
- Delete individual runs (with confirmation)
- Bulk remove all failed or completed runs
- Download report as HTML file (Content-Disposition: attachment)
- View + Download buttons in history table
- Backend: DELETE /api/runs/:id and DELETE /api/runs?status=failed|completed
- Backend: GET /report/:id/download serves with attachment header
- Updated both frontend/index.html and dashboard/index.html

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 13:23:56 -04:00
DJP
2473c22318 Fix Apache config: remove ProxyTimeout from Location block
ProxyTimeout is not allowed in <Location> context. Moved to server-level
ProxyTimeout directive already set above.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 12:18:02 -04:00
DJP
ce916cd658 Fix broken Unicode in Claude API calls + stabilize SSE proxy
- Sanitize unpaired surrogates from scraped text before JSON.stringify
  (Instagram captions often contain broken emoji causing JSON parse errors)
- Update Apache SSE proxy config: longer timeout, disable output filter
  buffering to prevent connection drops and repeated reconnects

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 12:15:07 -04:00
DJP
087d1bb23b Fix SSE reconnect loop: only POST /run once per pipeline start
EventSource auto-reconnects on connection drop, which re-fires the
'connected' event. The handler was POSTing /run on every reconnect,
causing multiple parallel pipeline runs and runaway Apify costs.

Added pipelineStarted guard so /run only fires on first connect.
Fixed in both frontend/index.html and dashboard/index.html.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:56:28 -04:00
DJP
9d15356a76 Fix Stage 5: correct actor input fields + add error resilience
- TikTok transcripts/comments actor expects 'videos' not 'videoUrls'
- Wrap all enrichment actor calls in safeRunActor so failures skip
  instead of crashing the pipeline

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:53:45 -04:00
DJP
57c4d3f0df Fix Apify budget: run scrapers sequentially instead of parallel
Promise.all() launched all platform scrapers simultaneously, so multiple
expensive runs started before any costs were tracked. Budget check only
saw totals after each run finished, allowing $7+ overspend on a $5 limit.

Now Stage 3 and Stage 5 run each scraper sequentially so the budget
gate can cut off between calls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:53:05 -04:00
DJP
247da45297 Fix port mapping: use env vars, bind localhost, remove duplicate ports
DB_PORT defaults to 5436, DASHBOARD_PORT defaults to 3456.
Prod override no longer redeclares ports (was causing duplicates).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:34:46 -04:00
DJP
9b9203355b Change prod Postgres port from 5435 to 5436 to avoid conflict
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:32:54 -04:00
DJP
c5c40aa4e5 Add server deployment: Apache proxy, static frontend, deploy script
- Static frontend (index.html, login.html, config.js) for Apache serving
- JSON-based auth API endpoints (/api/login, /api/auth, /api/logout)
- Apache config with ProxyPass for /social-reports path
- deploy/setup.sh for Ubuntu + Apache + Docker deployment
- docker-compose.prod.yml binds ports to 127.0.0.1 only
- Configurable API base URL via frontend/config.js

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:23:47 -04:00
DJP
ae981e8cb4 Add login auth, video embeds, and report serving
- Cookie-based session auth with login page (DASH_USER/DASH_PASS env vars)
- Serve generated reports via /report/:id route with View Report button
- YouTube iframe and Instagram native embeds in HTML reports
- Supporting videos grid per trend with platform icons
- Logout link in dashboard header

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 11:08:08 -04:00
DJP
50e1675b10 Initial commit: Social Listening Pipeline
8-stage TypeScript pipeline with Apify scraping, Claude AI analysis,
real-time dashboard with SSE, PostgreSQL cost tracking, and Apify budget controls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 22:18:02 -04:00
Dave Porter
d7b43dff99 Initial commit 2026-04-03 02:15:40 +00:00