modcomms/IVU_TESTING_PLAN.md

IVU Testing & Performance Monitoring Script

Context

Barclays IVU (Independent Validation Unit) Model compliance requires a structured methodology for testing and reporting on AI model performance. Per Section 7.1 of the requirements, the process involves ingesting 20 predefined assets, running them 5 times each, and comparing reports across Quality, Consistency, Accuracy, Completeness, and Response Time. This runs every 6 months to detect model drift and benchmark past responses.

This is a standalone Python CLI tool that connects to the running ModComms backend via WebSocket, runs batch analyses, scores results, and generates comparative reports. It does NOT live inside the site itself.

Directory Structure

testing/
├── ivu_runner.py              # Main CLI entry point
├── config.py                  # CLI args + env var configuration
├── models.py                  # Pydantic data models for IVU results
├── utils.py                   # CSV parsing, file I/O, MIME detection
├── ws_client.py               # Async WebSocket client
├── scoring/
│   ├── __init__.py
│   ├── ai_scorer.py           # Claude-based Quality/Accuracy/Completeness scorer (Anthropic API)
│   └── consistency_scorer.py  # Programmatic consistency comparison across runs
├── reporting/
│   ├── __init__.py
│   ├── pdf_report.py          # Per-run PDF report (reportlab)
│   ├── comparative_report.py  # Final HTML + PDF comparative summary
│   └── styles.py              # Shared brand colors and styling constants
├── assets/                    # Where the 20 test proof files go
│   └── .gitkeep
├── output/                    # Default output directory (gitignored)
│   └── .gitkeep
├── requirements.txt           # Standalone deps
└── sample_assets.csv          # Example CSV template

Implementation Steps

Step 1: Data Models (testing/models.py)

Define Pydantic models mirroring backend schemas from backend/app/models/schemas.py:

• SubReviewResult - mirrors SubReview (ragStatus, feedback, issues)
• AgentReviewResult - mirrors AgentReview (4 agent reviews + lead summary + overallStatus)
• SingleRunResult - one iteration result (agent_review, elapsed_seconds, iteration, error)
• AssetTestResult - all runs for one asset + computed scores
• IVUTestSuite - top-level model for the entire test run

Step 2: Configuration (testing/config.py)

argparse-based CLI with settings:

• --csv (required) - path to CSV file
• --assets-dir (default: ./assets) - directory containing proof files
• --backend-url (default: ws://localhost:8000/ws/analyze) - WebSocket endpoint
• --iterations (default: 5) - runs per asset
• --output-dir (default: ./output/{timestamp})
• --anthropic-api-key (from env ANTHROPIC_API_KEY) - for Claude-based scoring
• --access-token (default: ivu-test-token for DISABLE_AUTH=true)
• --delay-between-runs (default: 5 seconds)
• --baseline - path to previous results.json for drift comparison
• --skip-scoring / --skip-pdf flags

Step 3: CSV Parsing & Utilities (testing/utils.py)

CSV columns: campaign_name, brand_guidelines, client_lead, agency, agency_lead, proof_name, channel, sub_channel, proof_type, proof_file_name

Validates all required columns exist and all referenced files exist in assets directory before starting.

Step 4: WebSocket Client (testing/ws_client.py)

Async client using websockets library that:

• Reads file, base64-encodes it, detects MIME type
• Sends {"type": "analyze", ...} message (intentionally omits campaign_id and proof_name to avoid persisting test runs to the production database - see handlers.py:211)
• Listens for agent_started, agent_completed, model_fallback, complete, error messages
• Returns structured result with timing
• 30s connection timeout, 300s analysis timeout, 3 retries with exponential backoff on connection failures

Step 5: AI-Based Scoring (testing/scoring/ai_scorer.py)

Uses Claude (Anthropic API) as an independent judge to evaluate each run's output. This is a stronger IVU approach than using Gemini to score itself - a completely separate AI provider evaluates the analysis quality.

• Sends the proof image + agent review text to Claude with a detailed rubric
• Uses anthropic Python SDK with Claude Sonnet for cost-effective scoring
• Returns structured JSON scores (1-10) for:
  • Quality: Clarity, coherence, professional tone, no hallucinations, grammatical correctness
  • Accuracy: Information matches visible proof content, correct facts/numbers, appropriate RAG status
  • Completeness: All agent perspectives covered, legal/brand/channel aspects addressed, no omissions
• Requires ANTHROPIC_API_KEY env var (separate from the Gemini key used by the backend)

Step 6: Consistency Scoring (testing/scoring/consistency_scorer.py)

Programmatic comparison across iterations (no AI needed):

• RAG Status Consistency (25%): Unanimous RAG status across runs per agent
• Overall Status Consistency (25%): Same overallStatus across all runs
• Issue Overlap (25%): Fuzzy Jaccard similarity of issue lists using difflib.SequenceMatcher
• Feedback Similarity (25%): Normalized text comparison of feedback across runs

Step 7: Per-Run PDF Reports (testing/reporting/pdf_report.py)

Uses reportlab to generate PDFs matching the frontend's PDFReport.tsx layout:

• Cover page with Oliver brand, campaign info, iteration number
• Overall summary with RAG status and lead agent summary
• 4 agent review sections with RAG badges, feedback text, and key actions lists
• Brand colors: Navy #001f5a, Blue #00a3e0, RAG colors from PDFReport.tsx

Step 8: Comparative Report (testing/reporting/comparative_report.py)

Generates both HTML and PDF:

• Executive Summary: Run metadata, aggregate pass/fail rates, average scores
• Score Dashboard Table: One row per asset with avg Quality, Accuracy, Completeness, Consistency, Response Time
• Per-Asset Detail: Score breakdown by iteration, RAG distribution, response time stats
• Model Drift Analysis: If baseline provided, delta per asset per metric, flagged regressions (>1 point drop)

Step 9: Main Runner (testing/ivu_runner.py)

Orchestrates the full flow:

1. Parse CSV → validate files exist
2. For each asset (20):
   • For each iteration (5):
     • Run WebSocket analysis → collect result + timing
     • Generate per-run PDF
     • Score with AI scorer (Quality, Accuracy, Completeness)
     • Delay between runs
   • Compute consistency score across all iterations
3. Save raw results.json (for future baseline comparisons)
4. Generate comparative report (HTML + PDF)
5. Handle Ctrl+C gracefully (save partial results)

Output Structure

output/run_2026-03-03_14-30-00/
├── results.json                    # Machine-readable results (future baseline)
├── comparative_report.html         # Browser-viewable report
├── comparative_report.pdf          # PDF comparative report
├── per_run_pdfs/
│   ├── Asset01_ProofName_run1.pdf
│   ├── Asset01_ProofName_run2.pdf
│   └── ...                         # 20 assets × 5 runs = 100 PDFs
└── logs/
    └── ivu_runner.log              # Execution log

Dependencies (testing/requirements.txt)

websockets>=12.0
anthropic>=0.40.0
pydantic>=2.5.0
reportlab>=4.0

Key Files Referenced

• backend/app/websocket/handlers.py - WebSocket protocol (lines 42-58 for file handling, 211 for persistence gate)
• backend/app/models/schemas.py - Canonical SubReview/AgentReview schemas to mirror
• frontend/components/PDFReport.tsx - PDF layout, brand colors, and structure to replicate
• frontend/services/geminiService.ts - WebSocket client reference implementation
• backend/app/services/gemini_service.py - Gemini API client pattern (backend reference only)

Verification

1. Place test assets in testing/assets/, create a CSV with 2-3 assets
2. Start backend with DISABLE_AUTH=true
3. Run: cd testing && python ivu_runner.py --csv sample_assets.csv --iterations 2
4. Verify per-run PDFs are generated in output directory
5. Verify results.json contains all scores and timing data
6. Verify comparative report HTML opens in browser with correct data
7. Run again with --baseline output/{previous}/results.json to verify drift detection