agent_tracker/PLAN-prompt-audit.md

Agent Tracker: Prompt Audit & Auto-Audit — Implementation Plan

Context

The admin team needs visibility into what each LibreChat agent's system prompt says and whether it poses business/legal/compliance risks. System prompts will arrive via the agent-sync pipeline (see the agent-sync plan). Agent Tracker needs to: (1) accept and store them, and (2) provide on-demand AI analysis using Google Gemini to classify agents against business category definitions and flag those needing scrutiny.

Business Category Definitions (used for backend analysis, not visible fields)

• Cat 1: Created in the Oliver AI Sandbox, used behind the scenes for experimentation. Needs IT/Compliance consideration for safety.
• Cat 1B: Agents that may incur large cost but aren't Cat 2 or 3.
• Cat 2: Created in Pencil, exposed to clients but not for sale. Needs Legal consideration.
• Cat 3: Created in Pencil, sold to clients or sold via teams using them. Needs Commercial team consideration.

Architecture

Agent-sync pipeline → POST /agents collector API (includes instructions)
                              → Stored on agent document (instructions field)
                              → Auto-triggers classification for new/updated agents
                                    → Sends to Google Gemini API
                                    → Stores: discipline, department, category,
                                      risk_level, client detection, audit results
                                    → If client work detected:
                                        → Sets client = "yes"
                                        → Sets verification_status = "needs_verification"
                                        → Agent appears on Verification tab

Manual "Run Audit" button (admin dashboard)
       → Batch-processes all agents (or unclassified only)
       → Same Gemini analysis pipeline
       → Used for one-off catch-up of existing 600+ agents
       → Results displayed in Prompt Audit tab

• LLM: Google Gemini (gemini-2.5-pro) via google-genai SDK
• API Key: Reuses existing GOOGLE_API_KEY (same key as the ai_qc project)
• Trigger: Two modes:
  1. Automatic — runs after collector API creates/updates an agent (non-blocking background task)
  2. Manual — admin clicks "Run Audit" for batch processing or re-audit
• No new DB connections — prompts come through the existing collector API
• Field used: instructions (already stored on agent documents from LibreChat sync)

Files to Modify

1. MODIFY: requirements.txt

Add:

google-genai>=1.0.0

2. MODIFY: database.py — Add audit_history collection

After line 16 (after agent_ratings_collection), add:

audit_history_collection = db.get_collection("audit_history")

In ensure_indexes(), add:

await agents_collection.create_index([("audit_status", 1)])
await audit_history_collection.create_index([("agent_id", 1), ("audit_date", -1)])

Also update the import in audit_analyzer.py to use audit_history_collection.

3. MODIFY: models.py — Add audit models

Note: The instructions field already exists on AgentCollectorCreate, AiAgentCreate, AiAgent, and AiAgentResponse (added in 2026-03-26 update). No changes needed for that field.

Add to AiAgentResponse (around line 146, after instructions):

audit_status: Optional[str] = None
audit_date: Optional[str] = None
audit_category: Optional[str] = None
audit_risk_level: Optional[str] = None

Add new model after line 210:

class AuditReviewRequest(BaseModel):
    audit_status: str = Field(..., pattern="^(flagged|reviewed|cleared)$")
    reviewer_notes: Optional[str] = None

4. NEW: audit_analyzer.py — Core audit module

New file with two responsibilities: Gemini analysis and result storage.

A. Gemini API Analysis

import os
import json
import re
import asyncio
from datetime import datetime
from bson import ObjectId

import google.generativeai as genai

from database import agents_collection, audit_history_collection


def is_gemini_configured() -> bool:
    return bool(os.getenv("GOOGLE_API_KEY"))


def _get_model():
    api_key = os.getenv("GOOGLE_API_KEY")
    genai.configure(api_key=api_key)
    model_name = os.getenv("AUDIT_GEMINI_MODEL", "gemini-2.5-pro")
    return genai.GenerativeModel(model_name)

Key functions:

Function	Purpose
is_gemini_configured()	Checks GOOGLE_API_KEY env var
analyze_single_agent(agent_name, instructions, tools, description, model_name, author)	Builds prompt with category/discipline/client definitions, calls Gemini, parses JSON response
run_audit_batch(agents, concurrency=3)	asyncio.Semaphore for rate-limit-safe parallel processing
store_audit_result(agent_id, audit_data)	$set on agent doc (includes discipline, department, client flags) + insert into audit_history
apply_client_detection(agent_id, audit_data)	If is_client_work == true: sets client = "yes", verification_status = "needs_verification", and client_name if detected. Does NOT overwrite if agent already has client = "yes" set manually.
classify_single_agent(agent_id)	Convenience function for post-sync: loads agent doc, runs analyze_single_agent, stores result, applies client detection. Used by automatic trigger.
get_all_audit_results()	Query agents with audit fields
update_audit_review(agent_id, status, notes, reviewer_info)	Mark as reviewed/cleared

Gemini Prompt Design:

System instruction:

You are an AI agent compliance analyst and classifier. Analyse AI agents based on their
system prompts/instructions and metadata. You must:
1. Classify into business risk categories
2. Assign a business discipline
3. Infer the department/team from the instructions
4. Detect whether the agent is used for client-specific work

Respond ONLY with valid JSON matching this schema:
{
  "category": "1" | "1B" | "2" | "3",
  "category_reasoning": "string",
  "discipline": "Strategy" | "Creative" | "Oversight including delivery" | "Optimization" | "Back Office including operations" | "Pencil Agents",
  "discipline_reasoning": "string — why this discipline was chosen",
  "department": "string or null — inferred team/department from instructions (e.g. Project Management, Creative Services, Finance, Media, Strategy). null if not determinable",
  "is_client_work": true | false,
  "client_work_reasoning": "string — evidence from instructions that this agent handles client-specific work, references client names, brands, deliverables, or external-facing outputs. Empty string if not client work",
  "client_name_detected": "string or null — specific client or brand name found in instructions, null if none",
  "flags": ["array", "of", "strings"],
  "summary": "2-3 sentence analysis",
  "recommendations": "which team(s) should review and why",
  "risk_level": "low" | "medium" | "high" | "critical"
}

CATEGORY DEFINITIONS:
Cat 1 - Internal Sandbox/Experimentation (Oliver AI Sandbox, behind the scenes, needs IT/Compliance)
Cat 1B - High Cost Internal (may incur large cost, not Cat 2 or 3)
Cat 2 - Client-Exposed Not Sold (Pencil platform, exposed to clients, needs Legal)
Cat 3 - Client-Sold (Pencil platform, sold to clients, needs Commercial team)

DISCIPLINE DEFINITIONS (pick the best fit):
- Strategy: Agents focused on strategic planning, research, market analysis, insights
- Creative: Agents focused on creative work, content creation, design, copywriting
- Oversight including delivery: Agents focused on project management, delivery, QA, oversight, compliance
- Optimization: Agents focused on performance optimization, data analysis, efficiency, media planning
- Back Office including operations: Agents focused on internal operations, HR, finance, IT support, admin tasks
- Pencil Agents: Agents built on or for the Pencil platform specifically

DEPARTMENT INFERENCE:
Look for clues in the instructions about which team or role uses this agent. Examples:
- "I am a project manager" → department: "Project Management"
- "help the media team" → department: "Media"
- "creative brief" → department: "Creative"
- If no department/team clues found, set department to null

CLIENT WORK DETECTION:
An agent is client work (is_client_work = true) if the instructions reference:
- Specific client names or brand names
- Client deliverables, client presentations, client reports
- External-facing outputs meant for clients
- Client briefs, client feedback, client approvals
- Work explicitly described as "for the client" or "client-facing"
Do NOT flag as client work if the agent merely mentions "users" or "stakeholders" generically.

RISK LEVELS:
- low: Internal-only, limited capabilities
- medium: Internal with external tool access or moderate cost
- high: Client-facing or accesses sensitive data
- critical: Client-sold or handles financial/legal/PII data

FLAGS TO CONSIDER:
internal_only, experimental, sandbox, client_facing, pencil_platform,
revenue_generating, not_for_sale, uses_external_tools, uses_code_interpreter,
uses_file_search, accesses_sensitive_data, handles_pii, high_cost,
resource_intensive, legal_review_needed, commercial_review_needed,
compliance_review_needed, no_instructions

User prompt per agent:

Analyse this AI agent:

AGENT NAME: {name}
DESCRIPTION: {description}
MODEL: {model}
AUTHOR: {author}

INSTRUCTIONS (SYSTEM PROMPT):
---
{instructions}
---

TOOLS: {tools_list}
TOOL RESOURCES: {tool_resources}
ACTIONS: {actions}

Concurrency: asyncio.Semaphore(3) — configurable via AUDIT_CONCURRENCY env var. ~50 agents completes in ~50-85 seconds. For the one-off batch of 600+ agents, expect ~10-17 minutes at concurrency 3.

Error handling per agent: Wrap each call in try/except. On failure, return {"error": "...", "agent_name": "..."}. Parse JSON response with regex fallback if needed. All audited agents start with audit_status = "flagged".

Auto-classification post-sync: For single-agent classification triggered by the collector API, no semaphore needed — it's one call at a time. Uses classify_single_agent() which runs analyze_single_agent + store_audit_result + apply_client_detection in sequence.

One-off batch for existing agents: Use the POST /api/admin/audit/run endpoint (or a management script) to process all 600+ existing agents. The instructions field is already populated from LibreChat sync. Agents without instructions will be skipped (counted as skipped_count). The batch will:

1. Auto-assign discipline from the existing list
2. Infer department from the instructions
3. Classify into Cat 1/1B/2/3
4. Detect client work and auto-flag for verification
5. Store all results on agent documents

5. MODIFY: main.py — 3 new endpoints + automatic post-sync trigger

A. map_agent_collector_to_internal() — no change needed

The instructions field is already mapped through the collector API (added in the 2026-03-26 update).

B. Update create_agent_response() to include audit fields:

Add to the response construction:

audit_status=agent.get("audit_status"),
audit_date=agent.get("audit_date"),

C. Automatic post-sync classification trigger

In the collector API endpoint (POST /agents), after an agent is created or updated, trigger classification as a non-blocking background task:

from audit_analyzer import classify_single_agent, is_gemini_configured

# Inside the collector endpoint, after successful create/update:
if is_gemini_configured():
    # Fire-and-forget: classify in background, don't block the sync response
    asyncio.create_task(classify_single_agent(agent_id))

Key behaviours:

• Non-blocking: The collector API returns immediately; classification runs in background
• Idempotent: If the agent already has audit_status, re-classification overwrites with fresh results
• Graceful degradation: If GOOGLE_API_KEY not set, classification is silently skipped
• Client detection: If Gemini returns is_client_work = true, the background task auto-sets client = "yes" and verification_status = "needs_verification" on the agent doc (does NOT overwrite manually-set values)

D. Add 3 admin endpoints (after the analytics endpoint, ~line 1163):

Endpoint	Method	Purpose
POST /api/admin/audit/run	POST	Batch audit: reads stored instructions from agent docs, sends to Gemini. Optional agent_id body param for single agent. Optional unclassified_only param (default false) to only process agents without audit_status. Returns {status, total, audited_count, failed_count, skipped_count, results_summary}.
GET /api/admin/audit/results	GET	Returns all agents with audit data + config_status: {gemini_configured: bool}.
PUT /api/admin/audit/{agent_id}/review	PUT	Admin marks agent as reviewed/cleared with optional reviewer_notes.

Pre-flight check: POST /api/admin/audit/run returns 503 if GOOGLE_API_KEY not set.

6. MODIFY: templates/admin/dashboard.html — Add Prompt Audit tab

A. New tab (4th tab, after Agents Management):

<li class="nav-item">
  <a class="nav-link" id="audit-tab" data-bs-toggle="tab" href="#audit">
    <i class="fas fa-shield-alt me-2"></i>Prompt Audit
    <span class="badge bg-danger ms-1" id="auditFlaggedBadge" style="display:none;">0</span>
  </a>
</li>

B. Tab content layout:

Row 1: Header + [Run Audit] button (with spinner loading state) + [Run Unclassified Only] button
Row 2: Summary cards — Audited | Flagged | Reviewed | Cleared | Client Detected | No Instructions
Row 3: Filters — [Category ▼] [Risk Level ▼] [Discipline ▼] [Status ▼] [Search...]
Row 4: Results table:
        Agent Name | Discipline | Department | Category | Risk Level | Client Work | Flags | Status | Last Audited | Actions
Row 5: Agents without instructions notice (collapsible)

C. Audit Detail Modal:

• Agent name + basic info header
• LLM analysis summary
• Discipline badge + reasoning
• Department (inferred from instructions)
• Category badge + reasoning
• Client work detection: Yes/No badge + reasoning + detected client name (if any)
• Flags list (as badges)
• Recommendations text
• Instructions (collapsible <pre> block, can be long)
• Tools config (collapsible)
• Review controls: status dropdown (flagged/reviewed/cleared) + notes textarea + Save button
• Review history trail (who reviewed, when)

D. JavaScript functions:

Function	Purpose
loadAuditResults()	GET /api/admin/audit/results → populate table + summary cards + flagged badge count
runAudit(agentId?)	POST /api/admin/audit/run with loading spinner. Disable button during run. On complete, show summary alert + reload results.
displayAuditResults(agents)	Render table rows with color-coded badges
showAuditDetail(agentId)	Open detail modal with full analysis
submitAuditReview(agentId)	PUT /api/admin/audit/{id}/review → reload results
filterAuditResults()	Client-side filtering by category, risk, status, search text

Wire up:

• DOMContentLoaded → call loadAuditResults() alongside existing loadAdminData() and loadAnalytics()
• Refresh button → also call loadAuditResults()

E. CSS additions:

.risk-critical { background-color: #dc3545; color: white; }
.risk-high { background-color: #fd7e14; color: white; }
.risk-medium { background-color: #ffc107; color: #212529; }
.risk-low { background-color: #28a745; color: white; }

.cat-1 { background-color: #28a745; color: white; }
.cat-1b { background-color: #17a2b8; color: white; }
.cat-2 { background-color: #6f42c1; color: white; }
.cat-3 { background-color: #dc3545; color: white; }

.audit-status-flagged { border-left: 4px solid #dc3545; }
.audit-status-reviewed { border-left: 4px solid #ffc107; }
.audit-status-cleared { border-left: 4px solid #28a745; }

F. Config warning: If config_status.gemini_configured is false in the results response, show an info card explaining that GOOGLE_API_KEY needs to be set, and hide the Run Audit button.

Environment Variables

Add to .env:

GOOGLE_API_KEY=AIzaSyDMWN_PAnyU7bPmtWcEKq4LJfiu1KuwUsU
AUDIT_GEMINI_MODEL=gemini-2.5-pro     # default, configurable
AUDIT_CONCURRENCY=3                    # concurrent Gemini API calls

Error Handling

Scenario	Handling
GOOGLE_API_KEY not set	503 with message; UI shows config warning, hides Run button
Agent has no system_prompt	Skip during audit, count as "skipped", show in "No Prompt" card
Gemini API auth failure	Abort batch, return 503
Gemini rate limit	Retry with backoff; semaphore limits concurrency; mark agent as failed
Prompt too long	Gemini 2.5 Pro has 1M token limit — unlikely to hit; truncate at 900K chars if needed
JSON parse failure from Gemini	Regex extraction fallback \{[\s\S]*\}; if fails, store raw text as summary
Partial failures	Continue processing remaining agents; report success/fail/skipped counts

Design Decisions

• Instructions via agent-sync — instructions field already flows through the collector API, no second DB connection needed
• All audited agents start as "flagged" — admin must explicitly review/clear each
• Automatic post-sync classification — non-blocking asyncio.create_task() after collector API create/update, so sync is never slowed down
• Manual batch for catch-up — "Run Audit" button for the initial 600+ agent batch and future re-audits. Synchronous with spinner (~10-17 mins for full batch)
• Gemini 2.5 Pro — fast, 1M token context window, reuses existing Google API key
• Discipline auto-assignment — Gemini picks from the existing discipline list; written to the discipline field on the agent doc (overwrites only if currently empty, to respect manual edits)
• Department inference — free text field inferred from instructions; written to agent_department on the agent doc (overwrites only if currently empty)
• Client work auto-detection — Gemini scans instructions for client references; if detected, sets client = "yes" and verification_status = "needs_verification". Does NOT overwrite if client is already manually set to "yes" or "no".
• Separate from quality_audit_status — that's a manual human checkbox; this is automated analysis. They coexist.
• audit_history_collection — historical record of each audit run
• No keyword list initially — Gemini handles all client/discipline/department detection via LLM reasoning. Keyword overrides can be added later if needed.

Implementation Order

1. requirements.txt — add google-genai
2. database.py — add audit_history_collection + indexes
3. models.py — add audit fields to response model, add AuditReviewRequest
4. audit_analyzer.py — new file (Gemini analysis + result storage + client detection + discipline/department assignment)
5. main.py — update create_agent_response() + add 3 audit endpoints + add automatic post-sync trigger in collector API
6. templates/admin/dashboard.html — Prompt Audit tab (HTML, JS, CSS)
7. One-off batch run — deploy, then trigger POST /api/admin/audit/run to classify all 600+ existing agents

Verification

1. Add GOOGLE_API_KEY to .env
2. pip install -r requirements.txt
3. uvicorn main:app --reload --port 8000
4. Login as admin → /admin → click "Prompt Audit" tab
5. Agents with instructions show as ready for audit; agents without show in "No Instructions" count
6. Click "Run Audit" → spinner → results populate with category/risk/discipline/department/flags badges
7. Verify discipline was auto-assigned from the existing list
8. Verify department was inferred from instructions (or null if not determinable)
9. Check agents where is_client_work = true → confirm they now appear on the Verification tab with verification_status = "needs_verification"
10. Click detail on an agent → see full LLM analysis + raw instructions + client work reasoning
11. Mark as "Reviewed" with notes → status updates, reviewer trail shows
12. Test automatic trigger: POST a new agent via collector API → verify it auto-classifies within seconds
13. Refresh → all data persists
14. Test without GOOGLE_API_KEY → config warning shown gracefully, auto-trigger silently skipped
15. One-off batch: Run POST /api/admin/audit/run to process all 600+ existing agents, verify results

Files Changed Summary (for deployment)

File	Change
audit_analyzer.py	New file — Gemini analysis, discipline/department assignment, client detection, batch processing
database.py	Add audit_history_collection + indexes
models.py	Add audit fields to response model + AuditReviewRequest
main.py	3 new admin endpoints + automatic post-sync classification trigger in collector API + update create_agent_response()
templates/admin/dashboard.html	Add Prompt Audit tab with discipline/department/client columns
requirements.txt	Add google-genai

---

Phase 2: Read-Only Admin, Client Verification & Daily Digest

1. Read-Only Admin Role

Problem

Currently only two roles exist: user and admin. There's a need for users who can view the admin dashboard but not modify anything.

Solution

Add a third role: readonly_admin.

A. Data Model Changes (models.py)

• Update UserCreate and user handling to support role values: user, admin, readonly_admin

B. Auth Changes (auth.py / main.py)

• Add require_admin_or_readonly() dependency — allows access to admin dashboard GET routes
• Existing require_admin() stays as-is for write operations (create/update/delete)
• readonly_admin users:
  • Can access /admin dashboard and view all tabs
  • Cannot run audits, edit agents, edit users, approve verifications, or perform any write action
  • UI hides action buttons (edit, delete, approve, Run Audit) for readonly users

C. User Management UI (templates/admin/dashboard.html)

• In the user edit modal, add a role dropdown: User | Admin | Read-Only Admin
• Admin can change any user's role (except their own demotion from admin)

D. API Changes (main.py)

• GET /admin → allow admin + readonly_admin
• All POST/PUT/DELETE admin endpoints → require admin only (no change)
• GET /api/admin/* data endpoints → allow admin + readonly_admin
• New endpoint: PUT /api/users/{user_id}/role — admin-only, sets role

---

2. Client Verification System

Problem

Agents created for clients need a verification step before they're considered approved for use.

Solution

Add a verification_status field to agents and a verification workflow.

A. Data Model Changes (models.py)

• Add to AiAgent / AiAgentResponse:
  • client: Optional[str] = None — "yes" or "no"
  • client_name: Optional[str] = None — free text, required when client == "yes"
  • studio_name: Optional[str] = None — free text, not mandatory
  • verification_status: Optional[str] = None — "needs_verification" | "verified" | None
  • verified_by: Optional[str] = None — user who verified
  • verified_date: Optional[str] = None — when verified
• Add to AiAgentCreate:
  • client: str (mandatory, "yes" or "no")
  • client_name: Optional[str] = None
  • studio_name: Optional[str] = None
• Add to AgentCollectorCreate:
  • client: Optional[str] = None
  • client_name: Optional[str] = None
  • studio_name: Optional[str] = None

B. Registration Form (templates/agent_register.html)

Reorder form fields to:

1. Agent Name (mandatory)
2. Description (stays the same)
3. Purpose (stays the same)
4. Client — dropdown: Yes / No (mandatory)
   • 4a. If Yes → show a text input for Client Name (mandatory when visible)
5. Studio Name — free text (not mandatory)
6. Tool (stays the same)
7. ...rest of form continues from Version, Status, etc.

JavaScript: toggle client_name field visibility based on client dropdown value.

C. Auto-Tagging Logic

• When client == "yes" on agent creation → set verification_status = "needs_verification"
• When client == "no" or not set → verification_status remains None

D. Verification Tab on Admin Dashboard (templates/admin/dashboard.html)

New tab: Verification (between Agents Management and Prompt Audit tabs)

Tab content:
- Header: "Agents Pending Verification"
- Table: Agent Name | Client Name | Studio | Created By | Date Created | Status | Action
- Each row with status "needs_verification" shows an [Approve ✓] button
- Clicking Approve → calls PUT /api/admin/agents/{id}/verify → status changes to "verified"
- Verified agents move to a collapsible "Recently Verified" section below
- Filter: Show All | Needs Verification | Verified

E. API Endpoints (main.py)

• PUT /api/admin/agents/{agent_id}/verify — admin-only, sets verification_status = "verified", records verified_by and verified_date
• GET /api/admin/agents/pending-verification — returns agents where verification_status == "needs_verification"

F. Agent Card Display

• Show verification badge on agent cards:
  • needs_verification → orange badge: "Needs Verification"
  • verified → green badge: "Verified"
  • None → no badge (non-client agents)

---

3. Client Agent Email Notification

Problem

When a client-facing agent is created, stakeholders need to be notified immediately.

Solution

Trigger an email via Mailgun when client == "yes" on agent creation.

A. Mailgun Configuration (.env)

MAILGUN_API_KEY=4fccfbb0606b55852d243fc848f0356c-c6620443-faf31917
MAILGUN_DOMAIN=mg.oliver.solutions
MAILGUN_FROM_EMAIL=AgentHub <noreply@mg.oliver.solutions>
CLIENT_AGENT_NOTIFY_EMAILS=EstellevanHeerden@oliver.agency

Note: CLIENT_AGENT_NOTIFY_EMAILS is a comma-separated list, making it easy to add recipients later.

B. Notification Logic (notifications.py)

New function: send_client_agent_notification(agent_data: dict)

• Trigger: Called from the agent creation endpoint when client == "yes"
• Subject: Client Agent Created
• Body (HTML email):

  A new client-facing agent has been created and requires verification.
  
  Agent Name: {agent_name}
  Description: {description}
  Purpose: {purpose}
  Client: Yes
  Client Name: {client_name}
  Studio Name: {studio_name or "N/A"}
  Tool: {tool}
  Created By: {user_email}
  
  Please review this agent in AgentHub.
  

• To: CLIENT_AGENT_NOTIFY_EMAILS env var (comma-split)
• Non-blocking: Failure does not break agent creation (try/except, log warning)

C. Mailgun API Call Uses existing send_mailgun_email() pattern from notifications.py:

POST https://api.mailgun.net/v3/mg.oliver.solutions/messages
Auth: api:{MAILGUN_API_KEY}
Form data: from, to, subject, html

---

4. Daily Agent Digest Email

Problem

Admins need a daily summary of all agents created in the last 24 hours for quick scanning.

Solution

A scheduled daily email summarising new agents.

A. New Function (notifications.py)

send_daily_agent_digest()

• Query: Find all agents with created_date in the last 24 hours
• Subject: Agents Created Last 24 Hours
• Body (HTML email, kept short and scannable):

  {count} agent(s) created in the last 24 hours:
  
  ┌──────────────────────────────────────────────────┐
  │ 1. Agent Name                                    │
  │    Purpose: Brief purpose text                   │
  │    Description: Brief description text           │
  │    Created by: user@email.com                    │
  │                                                  │
  │ 2. Agent Name                                    │
  │    Purpose: ...                                  │
  │    Description: ...                              │
  │    Created by: ...                               │
  └──────────────────────────────────────────────────┘
  
  View full details at {AGENTHUB_URL}/admin
  

• To: All active admin users (query users collection for role == "admin")
• If no agents created: Skip sending (don't send empty digest)

B. Scheduling

Option: Use apscheduler (add to requirements.txt):

from apscheduler.schedulers.asyncio import AsyncIOScheduler

scheduler = AsyncIOScheduler()
scheduler.add_job(send_daily_agent_digest, 'cron', hour=7, minute=0)  # 7:00 AM daily
scheduler.start()

Wire into FastAPI startup event in main.py.

Alternatively, can be triggered via a cron job calling a new admin endpoint:

• POST /api/admin/digest/send — admin-only, manually trigger the digest

C. Configuration (.env)

DAILY_DIGEST_HOUR=7          # Hour to send (24h format), default 7
DAILY_DIGEST_TIMEZONE=UTC    # Timezone, default UTC

---

Implementation Order (Phase 2)

1. Models — Add client, client_name, studio_name, verification_status, verified_by, verified_date fields; update role to support readonly_admin
2. Database — Add index on verification_status
3. Auth — Add require_admin_or_readonly() dependency
4. Registration Form — Reorder fields, add Client dropdown with conditional Client Name, add Studio Name
5. CRUD/main.py — Auto-tag verification status on creation, add verification + role endpoints
6. Notifications — Client agent email + daily digest function
7. Admin Dashboard — Verification tab, role dropdown in user edit, readonly UI gating
8. Scheduler — Wire up daily digest (apscheduler or cron endpoint)

Files Changed Summary (Phase 2)

File	Change
models.py	Add client/verification/studio fields, readonly_admin role support
database.py	Add index on verification_status
auth.py	Add require_admin_or_readonly()
crud.py	Verification queries, daily digest query
main.py	Verification endpoints, role endpoint, digest endpoint, auth updates
notifications.py	Client agent notification, daily digest email
templates/agent_register.html	Reorder form, add Client/Studio fields
templates/admin/dashboard.html	Verification tab, role dropdown, readonly gating
requirements.txt	Add apscheduler (if using scheduler approach)
.env	Add CLIENT_AGENT_NOTIFY_EMAILS, DAILY_DIGEST_HOUR, DAILY_DIGEST_TIMEZONE

Verification (Phase 2)

1. Create a user with readonly_admin role → confirm they can view /admin but all action buttons are hidden
2. Register agent with Client = Yes, Client Name = "Test Client" → confirm verification_status = "needs_verification" and email sent to configured address
3. Register agent with Client = No → confirm no verification tag, no email
4. Admin dashboard → Verification tab → approve an agent → confirm status changes to "verified" with badge update
5. Wait for scheduled time (or trigger manually) → confirm daily digest email arrives with correct agent list
6. Test with no agents created → confirm no email sent

---

Dependency on Agent-Sync

The instructions field is already flowing through the agent-sync pipeline and stored on agent documents. No further sync changes are required. Agents without instructions will be skipped during audit (counted as "No Instructions") and can be re-audited once their instructions arrive via a future sync.

Flow Summary

┌─────────────────────────────────────────────────────────────────┐
│                        DAILY SYNC FLOW                          │
│                                                                 │
│  LibreChat ──sync──▶ POST /agents (collector API)              │
│                           │                                     │
│                           ▼                                     │
│                    Agent stored/updated in DB                   │
│                    (includes instructions field)                │
│                           │                                     │
│                           ▼                                     │
│              asyncio.create_task(classify_single_agent)         │
│                           │                                     │
│                           ▼                                     │
│                    Gemini Analysis                              │
│                    ├─ discipline (from defined list)            │
│                    ├─ department (free text, inferred)          │
│                    ├─ category (1/1B/2/3)                      │
│                    ├─ risk_level                                │
│                    ├─ is_client_work                            │
│                    └─ flags, summary, recommendations          │
│                           │                                     │
│                           ▼                                     │
│              Store results on agent doc                         │
│              If client work detected:                           │
│                ├─ client = "yes"                                │
│                ├─ verification_status = "needs_verification"    │
│                └─ Agent appears on Verification tab             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     ONE-OFF / MANUAL BATCH                      │
│                                                                 │
│  Admin clicks "Run Audit" on dashboard                         │
│       │                                                         │
│       ▼                                                         │
│  POST /api/admin/audit/run                                     │
│  (processes all agents, or unclassified_only=true)             │
│       │                                                         │
│       ▼                                                         │
│  Same Gemini pipeline as above, batched with semaphore(3)      │
│  ~600 agents ≈ 10-17 minutes                                   │
│       │                                                         │
│       ▼                                                         │
│  Results displayed in Prompt Audit tab                         │
│  Client agents flagged on Verification tab                     │
└─────────────────────────────────────────────────────────────────┘