---
auto_generated: true
created: 2026-05-18
manual_updated_at: 2026-05-18
modified: 2026-05-18
name: Developer_Manual
status: active
tags:
- domain/ai
- domain/devops
- tech/claude
- tech/docker
- tech/node
- tech/postgresql
- tech/typescript
type: sop
---

# Social Listening Pipeline: Developer Manual

## Architecture Overview

The project is a monorepo-style application consisting of:

- **Frontend**: Static HTML/JS served by Apache, handling UI and SSE connections.
- **Backend (Node.js)**: `agents/social-listening/dashboard/` handles HTTP API, pipeline orchestration, and database interactions.
- **Pipeline**: `agents/social-listening/pipeline-v2.ts` orchestrates 8 stages of data processing.
- **Database**: PostgreSQL (`db` service) stores run history, briefs, and cost logs.
- **Infrastructure**: Docker Compose manages dependencies (Postgres, App).

## Tech Stack

- **Runtime**: Node.js 20+
- **Database**: PostgreSQL 16
- **AI**: Anthropic Claude API (via `claude-cli.ts`)
- **Scraping**: Apify SDK
- **Frontend**: Vanilla JS + MSAL (Microsoft Authentication Library)
- **Deployment**: Docker & Docker Compose
- **Web Server**: Apache (for static frontend and reverse proxy)

## Local Setup

1. **Prerequisites**:
   - Docker & Docker Compose
   - Node.js 20+
   - Apify API Token
   - Anthropic API Key

2. **Configuration**:
   Copy `.env.example` to `.env` and fill in your keys:
   ```env
   APIFY_TOKEN=your_apify_token
   ANTHROPIC_API_KEY=your_anthropic_key
   APIFY_LIVE_APPROVED=true
   APIFY_COST_LIMIT=5
   TEST_MODE=false
   DASH_USER=admin
   DASH_PASS=changeme
   ```

3. **Run Services**:
   ```bash
   docker compose up -d
   ```

4. **Local Development (Optional)**:
   For live reload of the backend, run the Node.js server directly:
   ```bash
   cd agents/social-listening/dashboard
   npm install
   npm start
   ```
   Ensure `ALLOWED_ORIGIN` in `.env` permits `http://localhost:3456`.

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `APIFY_TOKEN` | Apify API token | - |
| `ANTHROPIC_API_KEY` | Anthropic API key | - |
| `APIFY_LIVE_APPROVED` | Enable live Apify actors | `false` |
| `APIFY_COST_LIMIT` | Max spend for Apify | `5` (USD) |
| `TEST_MODE` | Enable mock data | `false` |
| `DASHBOARD_PORT` | Port for the dashboard | `3456` |
| `DB_PORT` | Port for local DB access | `5436` |
| `DASH_USER` | Dashboard username | `admin` |
| `DASH_PASS` | Dashboard password | `changeme` |
| `ALLOWED_ORIGIN` | CORS allowed origin | Empty |
| `SESSION_SECRET` | Secret for session signing | - |
| `AZURE_TENANT_ID` | Azure AD Tenant ID | - |
| `AZURE_CLIENT_ID` | Azure AD Client ID | - |

## Key Services & Entry Points

### 1. Pipeline Orchestrator
- **File**: `agents/social-listening/pipeline-v2.ts`
- **Function**: `runPipeline(rawBrief, onProgress, onCost)`
- **Description**: Coordinates the 8-stage pipeline. Emits progress and cost updates via callbacks.
- **Stages**:
  1. `stage1-brief.js`: Validates input.
  2. `stage2-strategy-review.js`: AI strategy review.
  3. `stage3-discovery-scrape.js`: Apify scraping.
  4. `stage4-data-review.js`: AI trend analysis.
  5. `stage5-enrichment-scrape.js`: Fetches transcripts/metadata.
  6. `stage6-pre-report-review.js`: AI refinement.
  7. `stage7-desk-research.js`: Web search (if implemented).
  8. `stage8-report.js`: Generates final HTML.

### 2. Claude AI Client
- **File**: `agents/social-listening/claude-cli.ts`
- **Function**: `callClaude(messages, options)`
- **Description**: Wraps Anthropic API calls, handling token tracking and cost calculation. Uses `claude-opus-4-6` by default.

### 3. Database Layer
- **File**: `agents/social-listening/db.js`
- **Functions**: `createRun`, `logCostEvent`, `finishRun`, `getRunTotals`
- **Schema**: See `db/init.sql` for table definitions.

### 4. Frontend Configuration
- **File**: `frontend/config.js`
- **Variables**:
  - `window.__API_BASE`: Backend API path (`/social-reports`).
  - `window.__MSAL_CONFIG`: Azure AD SSO configuration.

## API Reference

The backend exposes REST endpoints under `/social-reports`:

- `POST /social-reports/run`: Start a new pipeline run.
- `GET /social-reports/runs`: List past runs.
- `GET /social-reports/runs/:id`: Get run details.
- `POST /social-reports/briefs`: Save a brief.
- `GET /social-reports/briefs`: List saved briefs.
- `DELETE /social-reports/briefs/:id`: Delete a brief.

*Note: Specific endpoint paths may vary; consult `dashboard/server.js` or equivalent for the exact router definitions.*

## Deployment

### Production Deployment
1. Use `docker-compose.prod.yml`:
   ```bash
   docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
   ```
2. Set `NODE_ENV=production` and a secure `SESSION_SECRET` in `.env`.
3. Ensure `ALLOWED_ORIGIN` is set to your domain.

### Azure AD SSO
1. Update `frontend/config.js` with your MSAL client ID and authority.
2. Set `AZURE_TENANT_ID` and `AZURE_CLIENT_ID` in environment variables.
3. Ensure the redirect URI is whitelisted in your Azure App Registration.

## Known Gotchas

- **Apify Actor Limits**: Ensure your Apify account has sufficient memory hours for scraping.
- **API Rate Limits**: Anthropic API has strict rate limits. Implement exponential backoff if needed.
- **CORS Issues**: If running frontend and backend separately, ensure `ALLOWED_ORIGIN` matches your frontend origin exactly.
- **Docker Volumes**: The `pgdata` volume persists DB data. Remove it to reset the database.
- **Unicode Handling**: The `claude-cli.ts` includes sanitization for unpaired surrogates to prevent JSON errors.

## Related
- [[agents/social-listening/pipeline-v2]]
- [[agents/social-listening/dashboard]]