ai_qc/backend/PROFILE_MANAGEMENT.md
nickviljoen 8bc1256e82 Add usage tracking reports, profile versioning, and token tracking
Implements three major feature enhancements:

1. Usage Tracking Reports
   - Command-line tool (generate_usage_report.py) for comprehensive usage reports
   - Supports text, JSON, and CSV output formats
   - Filters by date range, client, and user
   - Aggregates statistics by client, user, profile, and date
   - Automated report generation via cron jobs

2. Profile Auto-Versioning & Visibility Control
   - Automatic version control: edits create new versions (v2, v3, etc.)
   - Original profiles preserved for rollback capability
   - Profile visibility control (all clients vs client-specific)
   - Client-profile relationship management with dynamic updates
   - Audit trail with timestamps and user tracking

3. Actual Token Usage Tracking
   - Captures real token counts from OpenAI and Gemini APIs
   - Precise cost calculations instead of estimates (99% accuracy)
   - Per-check and per-provider token breakdowns
   - Pricing validation tool (validate_pricing.py)
   - Token usage optimization recommendations

Key Files Added:
- backend/generate_usage_report.py - Usage report generator
- backend/validate_pricing.py - Pricing validation tool
- backend/USAGE_REPORTS.md - Usage reports documentation
- backend/PROFILE_MANAGEMENT.md - Profile versioning guide
- backend/TOKEN_TRACKING_ENHANCEMENT.md - Token tracking guide
- backend/PRICING_GUIDE.md - Pricing validation guide
- backend/NEW_FEATURES_QUICKSTART.md - Quick start guide
- IMPLEMENTATION_SUMMARY.md - Complete implementation overview

Key Files Modified:
- backend/api_server.py - Profile versioning, token passthrough
- backend/client_config.py - Visibility-aware profile filtering
- backend/llm_config.py - Token usage extraction from APIs
- backend/usage_tracker.py - Actual token tracking and cost calculation
- CLAUDE.md - Updated documentation with new features

Benefits:
- Accurate cost tracking with real token usage
- Safe profile editing with version history
- Flexible profile visibility for multi-tenant setup
- Comprehensive usage analytics for optimization
- Better budget forecasting and client billing

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-02 13:22:33 +02:00

452 lines
11 KiB
Markdown

# Profile Management Guide
This guide explains the profile versioning and visibility features in the AI QC system.
## Features Overview
### 1. Automatic Profile Versioning
When you edit an existing profile, the system automatically:
- Creates a new version (v2, v3, etc.)
- Keeps the original profile unchanged
- Updates all client configurations to use the new version
### 2. Profile Visibility Control
When creating or editing profiles, you can control visibility:
- **All Clients**: Profile visible to all clients
- **Client-Specific**: Profile visible only to selected clients
## Profile Versioning
### How It Works
1. **Original Profile**: `my_profile.json` (version 1)
2. **First Edit**: Creates `my_profile_v2.json` (version 2)
3. **Second Edit**: Creates `my_profile_v3.json` (version 3)
4. And so on...
### Benefits
-**Safety**: Original profiles never modified
-**History**: Complete version history preserved
-**Rollback**: Easy to revert to previous versions
-**Audit Trail**: Track who made changes and when
-**Testing**: Test new versions without affecting production
### Version Metadata
Each profile version includes:
```json
{
"name": "My Profile",
"description": "Profile description",
"version": 2,
"created_at": "2026-02-02T10:00:00",
"created_by": "user@company.com",
"modified_at": "2026-02-02T15:30:00",
"modified_by": "admin@company.com",
"previous_version": "my_profile",
"visibility": "all",
"visible_to_clients": [],
"checks": { ... }
}
```
### Using Version History
#### View All Versions
```bash
# List all versions of a profile
ls -lh backend/profiles/my_profile*.json
# Example output:
# my_profile.json (original)
# my_profile_v2.json (first edit)
# my_profile_v3.json (second edit)
```
#### Compare Versions
```bash
# Compare original with v2
diff backend/profiles/my_profile.json backend/profiles/my_profile_v2.json
# View detailed changes
git diff --no-index backend/profiles/my_profile.json backend/profiles/my_profile_v2.json
```
#### Rollback to Previous Version
To rollback to a previous version, simply make it the "current" version by updating client configs:
1. Identify the version you want to use (e.g., `my_profile_v2.json`)
2. Edit `backend/client_config.py` to reference that version
3. Restart the application
Or use the API to "edit" the old version, which will create a new latest version with the old settings.
### API Endpoints
#### Create Profile (POST /api/profiles)
```bash
curl -X POST http://localhost:7183/api/profiles \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=YOUR_TOKEN" \
-d '{
"name": "My Custom Profile",
"description": "Profile for specific checks",
"visibility": "all",
"visible_to_clients": [],
"pass_threshold": 85,
"checks": {
"text_readability_general": {
"enabled": true,
"weight": 1.0,
"llm": "Gemini"
}
}
}'
```
Response:
```json
{
"status": "success",
"message": "Profile \"My Custom Profile\" created successfully",
"profile_id": "my_custom_profile",
"profile_path": "profiles/my_custom_profile.json",
"visibility": "all"
}
```
#### Update Profile (PUT /api/profiles/<profile_id>)
This automatically creates a new version:
```bash
curl -X PUT http://localhost:7183/api/profiles/my_custom_profile \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=YOUR_TOKEN" \
-d '{
"name": "My Custom Profile - Updated",
"description": "Updated profile description",
"visibility": "client_specific",
"visible_to_clients": ["diageo", "unilever"],
"checks": {
"text_readability_general": {
"enabled": true,
"weight": 1.0,
"llm": "Gemini"
},
"logo_visibility_general": {
"enabled": true,
"weight": 1.0,
"llm": "OpenAI"
}
}
}'
```
Response:
```json
{
"status": "success",
"message": "Profile updated to version 2. Original kept as \"my_custom_profile\"",
"profile_id": "my_custom_profile_v2",
"previous_profile_id": "my_custom_profile",
"version": 2
}
```
## Profile Visibility Control
### Visibility Options
#### 1. All Clients (Default)
Profile visible to all clients in the system.
```json
{
"visibility": "all",
"visible_to_clients": []
}
```
Use cases:
- General-purpose profiles
- Standard QC checks applicable to all brands
- Company-wide quality standards
#### 2. Client-Specific
Profile visible only to selected clients.
```json
{
"visibility": "client_specific",
"visible_to_clients": ["diageo", "unilever"]
}
```
Use cases:
- Brand-specific profiles
- Custom checks for specific clients
- Confidential QC criteria
### Client IDs
Available client IDs in the system:
- `diageo` - Diageo brand profiles
- `unilever` - Unilever brand profiles
- `loreal` - L'Oreal brand profiles
- `general` - General / Other clients
### How Visibility Works
1. **Profile Creation**: Set visibility during creation
2. **Client Filtering**: When users select a client, they only see profiles visible to that client
3. **Automatic Config Updates**: System automatically updates `client_config.py` based on visibility settings
4. **Dynamic Loading**: Profiles are loaded dynamically based on client context
### Examples
#### Example 1: Create Global Profile
```json
{
"name": "Universal Accessibility Check",
"description": "Accessibility checks for all clients",
"visibility": "all",
"visible_to_clients": [],
"checks": {
"accessibility": { "enabled": true, "weight": 1.0, "llm": "Gemini" },
"inclusive": { "enabled": true, "weight": 1.0, "llm": "Gemini" }
}
}
```
Result: Visible to all clients (diageo, unilever, loreal, general)
#### Example 2: Create Client-Specific Profile
```json
{
"name": "Diageo Premium Brand Check",
"description": "Premium brand guidelines for Diageo only",
"visibility": "client_specific",
"visible_to_clients": ["diageo"],
"checks": {
"logo_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" },
"product_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" },
"visual_hierarchy_general": { "enabled": true, "weight": 1.0, "llm": "Gemini" }
}
}
```
Result: Only visible when "Diageo" client is selected
#### Example 3: Multi-Client Profile
```json
{
"name": "Luxury Brand Standards",
"description": "Premium quality checks for luxury brands",
"visibility": "client_specific",
"visible_to_clients": ["diageo", "loreal"],
"checks": {
"visual_hierarchy_general": { "enabled": true, "weight": 1.0, "llm": "OpenAI" },
"element_alignment": { "enabled": true, "weight": 1.0, "llm": "Gemini" },
"product_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" }
}
}
```
Result: Visible to both Diageo and L'Oreal clients
## Web UI Integration
### Creating Profiles in Web UI
When creating a profile through the web interface, you'll see:
1. **Profile Name**: Enter profile name
2. **Description**: Describe the profile purpose
3. **Reveal to All Clients**: Checkbox
- ☑️ **Checked**: Profile visible to all clients
-**Unchecked**: Show client selector
4. **Client Selector** (if not revealed to all):
- Select specific clients that can use this profile
- Multiple clients can be selected
### Editing Profiles in Web UI
When editing a profile:
1. System automatically creates new version
2. Original profile remains unchanged
3. Success message shows: "Profile updated to version 2. Original kept as 'profile_name'"
4. New version becomes active in all client configurations
### Profile List View
Profiles in the list show:
- Profile name
- Version number (if versioned)
- Visibility status:
- 🌍 "All Clients" - Global visibility
- 🔒 "Specific Clients" - Limited visibility
- Client names listed for limited visibility
## Best Practices
### Profile Naming
-**Good**: `diageo_premium_2026`, `unilever_sustainability_check`
-**Avoid**: `test123`, `my_profile`, `new_new_final_v2`
### When to Use Versioning
- ✅ Changing check weights
- ✅ Modifying enabled checks
- ✅ Updating pass thresholds
- ✅ Changing profile descriptions
- ❌ Fixing typos (minor edits can be done manually if needed)
### When to Use Client-Specific Visibility
- ✅ Brand-specific QC criteria
- ✅ Confidential checks
- ✅ Client-requested custom profiles
- ✅ Experimental profiles for specific clients
- ❌ General-purpose checks
- ❌ Standard quality criteria
### Version Cleanup
Over time, you may accumulate many versions. Consider:
1. **Keep Recent Versions**: Keep last 3-5 versions
2. **Archive Old Versions**: Move to `profiles/archive/` folder
3. **Document Major Versions**: Add version notes in description
```bash
# Archive old versions (example)
mkdir -p backend/profiles/archive
mv backend/profiles/my_profile_v[1-3].json backend/profiles/archive/
```
## Troubleshooting
### Profile Not Appearing in Client Dropdown
Check:
1. Profile `visibility` setting in JSON file
2. Client ID in `visible_to_clients` array
3. `client_config.py` has been updated correctly
4. Browser cache (refresh page with Ctrl+F5)
### Version Number Not Incrementing
Check:
1. Profile was actually modified (not just re-saved)
2. No errors in browser console
3. Server logs for any errors
### Client Config Not Updating
Check:
1. Write permissions on `client_config.py`
2. Server logs for update errors
3. Manual review of `client_config.py` file
## Migration Guide
### Migrating Existing Profiles
Existing profiles without version/visibility fields will:
- Default to version 1
- Default to visibility "all"
- Continue working without changes
To add versioning to existing profiles:
```bash
# Add metadata to existing profile
python -c "
import json
profile_path = 'backend/profiles/my_profile.json'
with open(profile_path, 'r') as f:
profile = json.load(f)
profile['version'] = 1
profile['visibility'] = 'all'
profile['visible_to_clients'] = []
profile['created_at'] = '2026-02-02T10:00:00'
profile['created_by'] = 'system'
with open(profile_path, 'w') as f:
json.dump(profile, f, indent=2)
"
```
## API Reference
### Profile Schema
```typescript
interface Profile {
name: string; // Display name
description: string; // Profile description
version: number; // Version number (1, 2, 3, ...)
visibility: "all" | "client_specific"; // Visibility mode
visible_to_clients: string[]; // Client IDs (if client_specific)
created_at: string; // ISO datetime
created_by: string; // User email
modified_at?: string; // ISO datetime (if edited)
modified_by?: string; // User email (if edited)
previous_version?: string; // Previous version profile_id
pass_threshold: number; // Pass/fail threshold (0-100)
weight_scale?: number; // Optional weight scale
checks: {
[check_name: string]: {
enabled: boolean;
weight: number;
llm: "OpenAI" | "Gemini";
description?: string;
}
}
}
```
### Error Responses
```json
{
"status": "error",
"message": "Profile not found"
}
```
```json
{
"status": "error",
"message": "Profile already exists"
}
```
```json
{
"status": "error",
"message": "No changes detected"
}
```
## Support
For questions or issues:
1. Check profile JSON files in `backend/profiles/`
2. Review `backend/client_config.py` for client assignments
3. Check server logs for errors
4. Contact the development team