agent_tracker/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

AgentHub is a FastAPI-based AI Agent Management System with MongoDB backend. It provides three-tier role-based authentication (admin/readonly_admin/user), agent CRUD operations, client verification workflow, email notifications, user management, and a web interface built with Jinja2 templates and Bootstrap 5.

Common Development Tasks

Running the Application

uvicorn main:app --reload --port 8000

Access at: http://localhost:8000

Installing Dependencies

pip install -r requirements.txt

Environment Setup

Create .env file with:

• MONGODB_URI: MongoDB connection string (default: mongodb://localhost:27017)
• MONGODB_DBNAME: Database name (default: agenthub_db)
• SECRET_KEY: JWT secret key
• ALGORITHM: JWT algorithm (default: HS256)
• ACCESS_TOKEN_EXPIRE_MINUTES: Token expiration (default: 60)

Optional: Prompt Audit (Gemini)

• GOOGLE_API_KEY: Google API key for Gemini (if not set, audit is silently disabled)
• AUDIT_GEMINI_MODEL: Gemini model name (default: gemini-2.5-pro)
• AUDIT_CONCURRENCY: Batch size for sequential processing (default: 2)

Optional: Email Notifications (Mailgun)

• MAILGUN_API_KEY: Mailgun API key (if not set, notifications are silently disabled)
• MAILGUN_DOMAIN: Mailgun sending domain (e.g. your-domain.mailgun.org)
• MAILGUN_FROM_EMAIL: Sender address (default: AgentHub <noreply@{MAILGUN_DOMAIN}>)
• TOKEN_USAGE_THRESHOLD: Weekly token count that triggers an alert (default: 100000)
• NOTIFICATION_COOLDOWN_HOURS: Hours between repeat alerts for the same agent (default: 24)
• CLIENT_AGENT_NOTIFY_EMAILS: Comma-separated list of emails for client agent notifications
• WEEKLY_DIGEST_HOUR: Hour (24h format) to send weekly digest on Mondays (default: 7)

Default Login Credentials

• Admin: admin@agenthub.com / admin123
• Test User: test@example.com / testpass123

Code Architecture

Core Application Structure

main.py: FastAPI application with:

• JWT cookie-based authentication system
• HTML routes for web interface
• REST API endpoints for agent/user management
• Three-tier role-based access control (admin / readonly_admin / user)
• Client verification workflow with email notifications
• Daily agent digest scheduler (APScheduler)
• Template rendering with Jinja2

Key Authentication Functions:

• get_current_user_optional(): Cookie-based auth for templates
• get_current_user_from_cookie(): Required auth for API endpoints
• require_admin(): Admin-only access control (write operations)
• require_admin_or_readonly(): Admin + readonly_admin access (read-only dashboard views)

Data Layer

models.py: Pydantic models for:

• AiAgent: Core agent model with comprehensive fields (includes discipline, rating, client, client_name, studio_name)
• UsageTimelineEntry: Daily usage data including message_count and token_count
• UserCreate/UserResponse: User management models (includes role field: user/admin/readonly_admin)
• UserUpdate: Includes role field for three-tier role management
• AiAgentCreate/AiAgentResponse: API request/response models (includes total_tokens, prompt_tokens, completion_tokens, discipline, rating, rating_count, client, client_name, studio_name, verification_status, verified_by, verified_date)
• AgentCollectorCreate: Collector API input model (includes total_tokens, prompt_tokens, completion_tokens, discipline, client, client_name, studio_name)
• AuditReviewRequest: Audit review input (audit_status: flagged/reviewed/cleared, reviewer_notes)
• AgentUsageStatsResponse: Usage statistics response (includes total_tokens, prompt_tokens, completion_tokens)

crud.py: Database operations using Motor (async MongoDB driver):

• User CRUD: authentication, creation, management
• Agent CRUD: create, read, update, delete with user ownership
• Advanced features: search, filtering, statistics, pagination
• All operations use ObjectId for MongoDB document IDs

database.py: MongoDB connection setup with Motor async client

• Collections: users, agents, agent_usage, token_notifications, agent_ratings, audit_history
• ensure_indexes(): Creates compound unique index on agent_ratings(agent_id, user_id), indexes on verification_status, audit_status, and audit_history(agent_id, audit_date)

audit_analyzer.py: Gemini-powered agent classification and audit system:

• is_gemini_configured(): Returns False if GOOGLE_API_KEY not set (gracefully disabled)
• analyze_single_agent(): Sends agent instructions to Gemini 2.5 Pro, returns structured JSON with category, discipline, department, client detection, risk level, flags, and recommendations. Includes retry with exponential backoff for rate limits.
• store_audit_result(): Writes audit results to agent document (audit_status, audit_category, audit_risk_level, etc.) and inserts into audit_history collection
• apply_classification_fields(): Auto-assigns discipline (from defined list), agent_department (free text inferred from instructions), and client detection (client = "yes", verification_status = "needs_verification") — only overwrites fields that are currently empty/null
• classify_single_agent(): Convenience function for post-sync automatic classification. Loads agent, analyses, stores result, applies fields.
• run_audit_batch(): Batch processes agents sequentially with rate-limit-safe pauses. Supports unclassified_only and single_agent_id params. Returns summary with audited/failed/skipped counts.
• get_all_audit_results(): Returns all agents with audit fields for the Prompt Audit tab
• update_audit_review(): Admin marks audit as reviewed/cleared with notes
• Uses google-genai SDK (new API), model configurable via AUDIT_GEMINI_MODEL env var (default: gemini-2.5-pro)

notifications.py: Mailgun email notification system:

• is_mailgun_configured(): Returns False if env vars not set (gracefully disabled)
• send_mailgun_email(): POST to Mailgun HTTP API with 10s timeout
• build_threshold_email(): HTML email template for threshold alerts
• check_and_notify_threshold(): Checks 7-day token usage against threshold, enforces cooldown via token_notifications collection, sends to admin users
• send_client_agent_notification(): Sends email when client-facing agent is created (to CLIENT_AGENT_NOTIFY_EMAILS)
• build_client_agent_email(): HTML email template for client agent notifications
• send_weekly_agent_digest(): Queries agents created in last 7 days, sends summary to all admin users
• build_weekly_digest_email(): HTML email template for weekly digest

auth.py: JWT authentication with:

• bcrypt password hashing
• JWT token creation/validation using python-jose
• Configurable token expiration

Frontend Templates

Located in templates/ directory:

• base.html: Bootstrap 5 base template with navigation
• nav.html: Dynamic navigation based on user role
• index.html: Landing page
• login.html/register.html: Authentication forms
• agent_register.html: Agent creation form
• agent_management.html: Agent dashboard with real data
• search.html: Global search functionality
• user_management.html: User management interface
• admin/dashboard.html: Admin statistics, management, and verification workflow

Static Assets

static/style.css: Custom CSS with:

• CSS variables for consistent theming
• Gradient backgrounds and modern styling
• Responsive design for mobile devices
• Bootstrap 5 customizations

Key Features

Authentication Flow

• Cookie-based JWT authentication
• Three-tier role-based access: user, admin, readonly_admin
  • user: Standard access, can manage own agents
  • admin: Full access to admin dashboard, all write operations
  • readonly_admin: Can view admin dashboard but all write actions (edit, delete, approve, create) are hidden
• role field on user documents; is_admin kept in sync for backward compatibility
• require_admin_or_readonly() dependency for read-only admin endpoints
• Automatic redirects based on user role (admin/readonly_admin → /admin, user → /agent-management)
• Secure logout with token cleanup

Agent Management

• Full CRUD operations with user ownership
• Status tracking (Active, Inactive, Development, Deprecated)
• Rich metadata: tags, userbase, department, contact person
• Search functionality across multiple fields
• Filtering by status, audit status (Audited / Not Audited), and discipline
• Admin can view/manage all agents

Client & Verification System

• client field on agents: "yes" or "no" (mandatory on registration form)
• client_name: free text, required when client is "yes"
• studio_name: optional free text field
• Registration form order: Name, Description, Purpose, Client (Yes/No), Client Name (conditional), Studio Name, Tool, then Version/Status/etc.
• When client == "yes": agent auto-tagged with verification_status = "needs_verification"
• Verification tab on admin dashboard shows pending agents with Approve button
• PUT /api/admin/agents/{id}/verify — admin-only, sets status to "verified" with verifier info
• GET /api/admin/agents/pending-verification — returns agents needing verification
• Verification badges displayed on agent cards (orange "Needs Verification", green "Verified")

Client Agent Email Notification

• When client == "yes" on agent creation, sends email via Mailgun to CLIENT_AGENT_NOTIFY_EMAILS
• Subject: "Client Agent Created"
• Body includes: Agent Name, Description, Purpose, Client Name, Studio Name, Tool, Created By
• Non-blocking: failure does not break agent creation

Weekly Agent Digest Email

• Scheduled via APScheduler to run every Monday morning (default 7:00 AM, WEEKLY_DIGEST_HOUR env var)
• Queries agents created in last 7 days
• Sends to all active admin users
• Body includes: Agent Name, Purpose, Description, Created By (email)
• Subject: "Agents Created in Last Week"
• Skips sending if no agents created
• Can be manually triggered via POST /api/admin/digest/send (admin-only)

Discipline & Star Rating

• discipline field classifies agents into business categories: Strategy, Creative, Oversight including delivery, Optimization, Back Office including operations, Pencil Agents
• Required on registration form, optional on edit (to support legacy agents)
• Pencil Agents discipline is auto-assigned to agents with "pencil" in the name when no discipline is set (collector API auto-tag + startup migration)
• rating field stores the average star rating (1-5) computed from all per-user ratings
• rating_count field stores the number of individual ratings
• Per-user rating system: Any authenticated user can rate any agent via PUT /api/agents/{id}/rating
  • Individual ratings stored in agent_ratings collection with compound unique index on (agent_id, user_id)
  • After each rating, the agent's average rating and count are recalculated and stored on the agent document
  • GET /api/agents/{id}/my-rating returns the current user's rating plus the average and count
• Interactive star rating widget in detail modal shows the user's own rating as filled stars
• Average rating and count displayed below the stars and on agent card badges
• Rating removed from edit modals (rating is per-user, not admin-set)
• Rating framework info modal accessible via info icon next to "Rating:" label
• Dashboard supports filtering by discipline and sorting by rating
• Discipline badge (purple) and star rating badge displayed on agent cards
• Both fields included in CSV export/import
• Discipline passed through collector API; rating is human-only (not in collector)

Token Usage Tracking

• total_tokens, prompt_tokens, completion_tokens fields on agents track cumulative LLM token consumption
• prompt_tokens (input) and completion_tokens (output) provide cost breakdown detail
• token_count per day in usage timeline entries alongside message_count
• Token badge displayed on agent cards (gold/coins icon) with prompt/completion breakdown in tooltip
• Usage modal shows Total Tokens stat with In/Out breakdown alongside messages/conversations/users
• Dual-axis chart (messages left axis, tokens right axis) when token data exists
• Sort agents by Total Tokens
• CSV export includes total_tokens, prompt_tokens, completion_tokens columns

High Usage Email Notifications

• Entirely optional — silently disabled when Mailgun env vars are not set
• Triggered from the Agent Collector endpoint (POST /agents); checks 7-day rolling token usage from usage_timeline
• Alerts when weekly token usage exceeds threshold (default 100,000, configurable via TOKEN_USAGE_THRESHOLD)
• Non-blocking — notification failure never breaks the collector API
• Cooldown tracking in MongoDB token_notifications collection (default 24h, configurable)
• Sends to all active admin users' email addresses

Prompt Audit & Auto-Classification (Gemini)

• Automated analysis of agent instructions (system prompts) using Google Gemini 2.5 Pro
• Two trigger modes:
  1. Automatic post-sync: After the collector API (POST /agents) creates/updates an agent with instructions, a background task (asyncio.create_task) auto-classifies it. Non-blocking — sync response is not delayed.
  2. Manual batch: Admin clicks "Run Full Audit" or "Run Unclassified Only" on the Prompt Audit tab. Processes agents sequentially with 4-second pauses between batches to avoid Gemini rate limits.
• Classification outputs per agent:
  • audit_category: Cat 1 (Internal Sandbox), Cat 1B (High Cost Internal), Cat 2 (Client-Exposed), Cat 3 (Client-Sold)
  • audit_risk_level: low / medium / high / critical
  • audit_discipline: Picks from existing discipline list (Strategy, Creative, Oversight including delivery, Optimization, Back Office including operations, Pencil Agents)
  • audit_department: Free text inferred from instructions (e.g., "Project Management", "Media")
  • audit_is_client_work: Boolean — detects client names, brands, client deliverables in instructions
  • audit_flags: Array of risk flags (client_facing, handles_pii, uses_external_tools, etc.)
  • audit_summary, audit_recommendations, audit_category_reasoning, audit_discipline_reasoning, audit_client_work_reasoning, audit_client_name_detected
• Auto-assignment: Gemini results auto-populate discipline and agent_department fields on the agent document (only if currently empty, to respect manual edits)
• Client work auto-detection: When audit_is_client_work = true, auto-sets client = "yes" and verification_status = "needs_verification" — agent appears on Verification tab. Does NOT overwrite manually-set values.
• Audit statuses: All audited agents start as flagged. Admin can mark as reviewed or cleared via the detail modal with optional notes.
• Prompt Audit tab on admin dashboard: Summary cards (Audited, Flagged, Reviewed, Cleared, Client Detected, No Instructions), filterable results table, detail modal with full analysis and review controls
• Rate limit handling: Retry with exponential backoff (10s, 20s, 40s) for 429/quota errors
• Logging: Uses Python logging module (audit_analyzer logger) for systemd journal visibility
• Agents without instructions are skipped (counted as "No Instructions")
• audit_history collection stores historical record of each audit run and review action
• Gracefully disabled when GOOGLE_API_KEY not set — UI shows config warning, buttons hidden

User Management

• User registration with validation
• Admin user creation capabilities
• Three-tier role system: user, admin, readonly_admin
  • Role dropdown in admin user edit modal (replaces is_admin checkbox)
  • role and is_admin fields kept in sync on update
• Profile management
• User statistics and administration

Database Integration

• MongoDB with proper ObjectId handling
• Async operations using Motor driver
• Indexed queries for performance
• Data aggregation for statistics
• Collections: users, agents, agent_usage, token_notifications, agent_ratings, audit_history

Development Guidelines

Database Operations

• Always use ObjectId for MongoDB document IDs
• Use Motor async driver methods (await collection.find_one())
• Handle ObjectId conversion in CRUD operations
• Implement proper error handling with try/except blocks

Authentication

• Use cookie-based auth for web interface
• API endpoints require get_current_user_from_cookie() dependency
• Write endpoints use require_admin() dependency
• Read-only admin endpoints use require_admin_or_readonly() dependency
• Always validate user permissions for data access
• readonly_admin users can view admin dashboard but UI hides all write-action buttons via admin-write-action CSS class

Template Context

• Pass current_user to all templates for navigation
• Handle dict objects (not User model instances) in templates
• Use proper null checks for optional user data

API Response Models

• Convert ObjectId to string in API responses
• Handle optional datetime fields with isoformat()
• Maintain consistency between Create and Response models

Error Handling

• Provide meaningful error messages in templates
• Use proper HTTP status codes in API responses
• Graceful degradation for missing data

Project Dependencies

Key dependencies from requirements.txt:

• fastapi: Web framework
• uvicorn: ASGI server
• motor: Async MongoDB driver
• pymongo: MongoDB operations
• python-jose: JWT token handling
• passlib: Password hashing
• bcrypt: Password encryption
• pydantic: Data validation
• jinja2: Template engine
• python-multipart: Form handling
• requests: HTTP client (used for Mailgun API calls)
• apscheduler: Task scheduling (weekly digest email)
• google-genai: Google Gemini API client (used for prompt audit auto-classification)

API Endpoints (New)

Verification

• GET /api/admin/agents/pending-verification — List agents with verification status (admin + readonly_admin)
• PUT /api/admin/agents/{agent_id}/verify — Approve/verify an agent (admin only)

Weekly Digest

• POST /api/admin/digest/send — Manually trigger the weekly agent digest email (admin only)

Prompt Audit

• POST /api/admin/audit/run — Run Gemini audit batch. Optional JSON body: {agent_id, unclassified_only}. Returns {status, total, audited_count, failed_count, skipped_count, results_summary} (admin only)
• GET /api/admin/audit/results — Get all agents with audit data + config_status.gemini_configured (admin + readonly_admin)
• PUT /api/admin/audit/{agent_id}/review — Mark audit as reviewed/cleared with {audit_status, reviewer_notes} (admin only)