Oliver/social-reporting-tool
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
17 commits 2 branches 0 tags 740 KiB
Find a file
DJP f2d6f56831 Report quality overhaul: 11 feedback items 
1. Remove Desk Research (Stage 7 skipped, sources removed from report)
2. Fix comments scraping: increase cap to 2000, handle alt field names
3. Dynamic stats bar: hide zero-value stats instead of showing "0 Comments"
4. Prompt improvements: enforce timeliness, comment-based insights, creator spotlight algorithm (2-10 videos, exclude >50% dominance)
5. Date filtering: pass date params to Apify actors (oldestCreateTime, onlyPostsNewerThan, uploadDate) + log filter counts
6. Pullquotes: 3-4 generated editorial dividers between sections
7. Thumbnails: download top 50 coverUrl as base64, store on EnrichedVideo
8. Visual Language section: 5 batches of 10 through Claude Vision, synthesized into 5-6 visual codes with thumbnail cards
9. Sticky navigation bar with anchor links to all sections
10. New types: VisualCode, thumbnailUrl on Video, thumbnailBase64 on EnrichedVideo, pullquotes/visualCodes on ReportJSON

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 09:52:08 -04:00
agents/social-listening Report quality overhaul: 11 feedback items 2026-04-08 09:52:08 -04:00
db Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
deploy Fix Apache config: remove ProxyTimeout from Location block 2026-04-07 12:18:02 -04:00
frontend Round cost displays to nearest cent 2026-04-07 13:43:41 -04:00
.gitignore Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
DEVELOPER_BRIEF.md Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
docker-compose.prod.yml Fix port mapping: use env vars, bind localhost, remove duplicate ports 2026-04-07 11:34:46 -04:00
docker-compose.yml Fix port mapping: use env vars, bind localhost, remove duplicate ports 2026-04-07 11:34:46 -04:00
Dockerfile Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
package-lock.json Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
package.json Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00
README.md Add project README with architecture, setup, and deployment docs 2026-04-07 14:13:24 -04:00
tsconfig.json Initial commit: Social Listening Pipeline 2026-04-02 22:18:02 -04:00

README.md

Social Listening Pipeline

Automated social media research tool that scrapes TikTok, Instagram, and YouTube via Apify, analyses content with Claude AI, and generates client-ready HTML reports.

Architecture

frontend/          Static frontend (served by Apache)
agents/social-listening/
  dashboard/       Node.js backend (HTTP + SSE on port 3456)
  stages/          8-stage pipeline
  briefs/          Saved client briefs (JSON)
  outputs/         Generated reports
deploy/            Apache config + setup script

Pipeline Stages

Stage Name Description
1 Brief Validation Validates and normalises the client brief
2 Strategy Review AI reviews strategy, suggests up to 3 extra hashtags
3 Discovery Scrape Scrapes TikTok/Instagram/YouTube via Apify
4 Data Review AI analyses scraped content for trends
5 Enrichment Scrape Fetches transcripts and extra metadata
6 Pre-Report Review AI refines findings before report generation
7 Desk Research Web search for additional context
8 Report Generation Produces final HTML report with video embeds

Key Features

  • Real-time dashboard with SSE progress updates and live cost tracking
  • Apify budget control (APIFY_COST_LIMIT) — stops scraping when limit is reached
  • Saved briefs — save/load client briefs server-side with a dedicated tab
  • Run history — view, download, and delete past pipeline runs with cost breakdowns
  • Video embeds — YouTube iframes, Instagram native embeds, TikTok links in reports
  • Auth — cookie-based session auth with HMAC-signed tokens

Prerequisites

  • Docker & Docker Compose
  • Node.js 20+ (for local development)
  • Apify API token
  • Anthropic API key

Environment Variables

Copy .env.example or create .env in the project root:

APIFY_TOKEN=your_apify_token
ANTHROPIC_API_KEY=your_anthropic_key
APIFY_LIVE_APPROVED=true
APIFY_COST_LIMIT=5
TEST_MODE=false
DASHBOARD_PORT=3456
DATABASE_URL=postgres://social:social@db:5432/social_listening
DASH_USER=admin
DASH_PASS=changeme
SESSION_SECRET=random_secret_here

Running Locally

# Start PostgreSQL + app via Docker
docker compose up -d

# Dashboard available at http://localhost:3456

Or without Docker:

npm install

# Start the dashboard server
npm run dashboard

# Run pipeline directly (CLI)
npm run pipeline        # dry run
npm run pipeline:test   # test mode
npm run pipeline:live   # live Apify scraping

Production Deployment

The app is designed to run behind Apache on an Ubuntu server:

  • Backend: Docker containers at /opt/social-reporting
  • Frontend: Static files at /var/www/html/social-reporting
  • URL: https://your-domain.com/social-reports/
# On the server
cd /opt/social-reporting
git pull
cp frontend/* /var/www/html/social-reporting/
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build

See deploy/apache-social-reports.conf for the Apache reverse proxy config and deploy/setup.sh for first-time setup.

Tech Stack

  • Runtime: TypeScript (ESM) via tsx
  • Backend: Node.js HTTP server with SSE
  • Database: PostgreSQL (via postgres npm package)
  • Scraping: Apify REST API
  • AI: Anthropic Claude API (Messages API)
  • Frontend: Vanilla HTML/CSS/JS with Montserrat font
  • Deploy: Docker Compose + Apache reverse proxy