Implements three major feature enhancements: 1. Usage Tracking Reports - Command-line tool (generate_usage_report.py) for comprehensive usage reports - Supports text, JSON, and CSV output formats - Filters by date range, client, and user - Aggregates statistics by client, user, profile, and date - Automated report generation via cron jobs 2. Profile Auto-Versioning & Visibility Control - Automatic version control: edits create new versions (v2, v3, etc.) - Original profiles preserved for rollback capability - Profile visibility control (all clients vs client-specific) - Client-profile relationship management with dynamic updates - Audit trail with timestamps and user tracking 3. Actual Token Usage Tracking - Captures real token counts from OpenAI and Gemini APIs - Precise cost calculations instead of estimates (99% accuracy) - Per-check and per-provider token breakdowns - Pricing validation tool (validate_pricing.py) - Token usage optimization recommendations Key Files Added: - backend/generate_usage_report.py - Usage report generator - backend/validate_pricing.py - Pricing validation tool - backend/USAGE_REPORTS.md - Usage reports documentation - backend/PROFILE_MANAGEMENT.md - Profile versioning guide - backend/TOKEN_TRACKING_ENHANCEMENT.md - Token tracking guide - backend/PRICING_GUIDE.md - Pricing validation guide - backend/NEW_FEATURES_QUICKSTART.md - Quick start guide - IMPLEMENTATION_SUMMARY.md - Complete implementation overview Key Files Modified: - backend/api_server.py - Profile versioning, token passthrough - backend/client_config.py - Visibility-aware profile filtering - backend/llm_config.py - Token usage extraction from APIs - backend/usage_tracker.py - Actual token tracking and cost calculation - CLAUDE.md - Updated documentation with new features Benefits: - Accurate cost tracking with real token usage - Safe profile editing with version history - Flexible profile visibility for multi-tenant setup - Comprehensive usage analytics for optimization - Better budget forecasting and client billing Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
452 lines
11 KiB
Markdown
452 lines
11 KiB
Markdown
# Profile Management Guide
|
|
|
|
This guide explains the profile versioning and visibility features in the AI QC system.
|
|
|
|
## Features Overview
|
|
|
|
### 1. Automatic Profile Versioning
|
|
When you edit an existing profile, the system automatically:
|
|
- Creates a new version (v2, v3, etc.)
|
|
- Keeps the original profile unchanged
|
|
- Updates all client configurations to use the new version
|
|
|
|
### 2. Profile Visibility Control
|
|
When creating or editing profiles, you can control visibility:
|
|
- **All Clients**: Profile visible to all clients
|
|
- **Client-Specific**: Profile visible only to selected clients
|
|
|
|
## Profile Versioning
|
|
|
|
### How It Works
|
|
|
|
1. **Original Profile**: `my_profile.json` (version 1)
|
|
2. **First Edit**: Creates `my_profile_v2.json` (version 2)
|
|
3. **Second Edit**: Creates `my_profile_v3.json` (version 3)
|
|
4. And so on...
|
|
|
|
### Benefits
|
|
|
|
- ✅ **Safety**: Original profiles never modified
|
|
- ✅ **History**: Complete version history preserved
|
|
- ✅ **Rollback**: Easy to revert to previous versions
|
|
- ✅ **Audit Trail**: Track who made changes and when
|
|
- ✅ **Testing**: Test new versions without affecting production
|
|
|
|
### Version Metadata
|
|
|
|
Each profile version includes:
|
|
|
|
```json
|
|
{
|
|
"name": "My Profile",
|
|
"description": "Profile description",
|
|
"version": 2,
|
|
"created_at": "2026-02-02T10:00:00",
|
|
"created_by": "user@company.com",
|
|
"modified_at": "2026-02-02T15:30:00",
|
|
"modified_by": "admin@company.com",
|
|
"previous_version": "my_profile",
|
|
"visibility": "all",
|
|
"visible_to_clients": [],
|
|
"checks": { ... }
|
|
}
|
|
```
|
|
|
|
### Using Version History
|
|
|
|
#### View All Versions
|
|
|
|
```bash
|
|
# List all versions of a profile
|
|
ls -lh backend/profiles/my_profile*.json
|
|
|
|
# Example output:
|
|
# my_profile.json (original)
|
|
# my_profile_v2.json (first edit)
|
|
# my_profile_v3.json (second edit)
|
|
```
|
|
|
|
#### Compare Versions
|
|
|
|
```bash
|
|
# Compare original with v2
|
|
diff backend/profiles/my_profile.json backend/profiles/my_profile_v2.json
|
|
|
|
# View detailed changes
|
|
git diff --no-index backend/profiles/my_profile.json backend/profiles/my_profile_v2.json
|
|
```
|
|
|
|
#### Rollback to Previous Version
|
|
|
|
To rollback to a previous version, simply make it the "current" version by updating client configs:
|
|
|
|
1. Identify the version you want to use (e.g., `my_profile_v2.json`)
|
|
2. Edit `backend/client_config.py` to reference that version
|
|
3. Restart the application
|
|
|
|
Or use the API to "edit" the old version, which will create a new latest version with the old settings.
|
|
|
|
### API Endpoints
|
|
|
|
#### Create Profile (POST /api/profiles)
|
|
|
|
```bash
|
|
curl -X POST http://localhost:7183/api/profiles \
|
|
-H "Content-Type: application/json" \
|
|
-H "Cookie: auth_token=YOUR_TOKEN" \
|
|
-d '{
|
|
"name": "My Custom Profile",
|
|
"description": "Profile for specific checks",
|
|
"visibility": "all",
|
|
"visible_to_clients": [],
|
|
"pass_threshold": 85,
|
|
"checks": {
|
|
"text_readability_general": {
|
|
"enabled": true,
|
|
"weight": 1.0,
|
|
"llm": "Gemini"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"status": "success",
|
|
"message": "Profile \"My Custom Profile\" created successfully",
|
|
"profile_id": "my_custom_profile",
|
|
"profile_path": "profiles/my_custom_profile.json",
|
|
"visibility": "all"
|
|
}
|
|
```
|
|
|
|
#### Update Profile (PUT /api/profiles/<profile_id>)
|
|
|
|
This automatically creates a new version:
|
|
|
|
```bash
|
|
curl -X PUT http://localhost:7183/api/profiles/my_custom_profile \
|
|
-H "Content-Type: application/json" \
|
|
-H "Cookie: auth_token=YOUR_TOKEN" \
|
|
-d '{
|
|
"name": "My Custom Profile - Updated",
|
|
"description": "Updated profile description",
|
|
"visibility": "client_specific",
|
|
"visible_to_clients": ["diageo", "unilever"],
|
|
"checks": {
|
|
"text_readability_general": {
|
|
"enabled": true,
|
|
"weight": 1.0,
|
|
"llm": "Gemini"
|
|
},
|
|
"logo_visibility_general": {
|
|
"enabled": true,
|
|
"weight": 1.0,
|
|
"llm": "OpenAI"
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
```json
|
|
{
|
|
"status": "success",
|
|
"message": "Profile updated to version 2. Original kept as \"my_custom_profile\"",
|
|
"profile_id": "my_custom_profile_v2",
|
|
"previous_profile_id": "my_custom_profile",
|
|
"version": 2
|
|
}
|
|
```
|
|
|
|
## Profile Visibility Control
|
|
|
|
### Visibility Options
|
|
|
|
#### 1. All Clients (Default)
|
|
|
|
Profile visible to all clients in the system.
|
|
|
|
```json
|
|
{
|
|
"visibility": "all",
|
|
"visible_to_clients": []
|
|
}
|
|
```
|
|
|
|
Use cases:
|
|
- General-purpose profiles
|
|
- Standard QC checks applicable to all brands
|
|
- Company-wide quality standards
|
|
|
|
#### 2. Client-Specific
|
|
|
|
Profile visible only to selected clients.
|
|
|
|
```json
|
|
{
|
|
"visibility": "client_specific",
|
|
"visible_to_clients": ["diageo", "unilever"]
|
|
}
|
|
```
|
|
|
|
Use cases:
|
|
- Brand-specific profiles
|
|
- Custom checks for specific clients
|
|
- Confidential QC criteria
|
|
|
|
### Client IDs
|
|
|
|
Available client IDs in the system:
|
|
- `diageo` - Diageo brand profiles
|
|
- `unilever` - Unilever brand profiles
|
|
- `loreal` - L'Oreal brand profiles
|
|
- `general` - General / Other clients
|
|
|
|
### How Visibility Works
|
|
|
|
1. **Profile Creation**: Set visibility during creation
|
|
2. **Client Filtering**: When users select a client, they only see profiles visible to that client
|
|
3. **Automatic Config Updates**: System automatically updates `client_config.py` based on visibility settings
|
|
4. **Dynamic Loading**: Profiles are loaded dynamically based on client context
|
|
|
|
### Examples
|
|
|
|
#### Example 1: Create Global Profile
|
|
|
|
```json
|
|
{
|
|
"name": "Universal Accessibility Check",
|
|
"description": "Accessibility checks for all clients",
|
|
"visibility": "all",
|
|
"visible_to_clients": [],
|
|
"checks": {
|
|
"accessibility": { "enabled": true, "weight": 1.0, "llm": "Gemini" },
|
|
"inclusive": { "enabled": true, "weight": 1.0, "llm": "Gemini" }
|
|
}
|
|
}
|
|
```
|
|
|
|
Result: Visible to all clients (diageo, unilever, loreal, general)
|
|
|
|
#### Example 2: Create Client-Specific Profile
|
|
|
|
```json
|
|
{
|
|
"name": "Diageo Premium Brand Check",
|
|
"description": "Premium brand guidelines for Diageo only",
|
|
"visibility": "client_specific",
|
|
"visible_to_clients": ["diageo"],
|
|
"checks": {
|
|
"logo_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" },
|
|
"product_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" },
|
|
"visual_hierarchy_general": { "enabled": true, "weight": 1.0, "llm": "Gemini" }
|
|
}
|
|
}
|
|
```
|
|
|
|
Result: Only visible when "Diageo" client is selected
|
|
|
|
#### Example 3: Multi-Client Profile
|
|
|
|
```json
|
|
{
|
|
"name": "Luxury Brand Standards",
|
|
"description": "Premium quality checks for luxury brands",
|
|
"visibility": "client_specific",
|
|
"visible_to_clients": ["diageo", "loreal"],
|
|
"checks": {
|
|
"visual_hierarchy_general": { "enabled": true, "weight": 1.0, "llm": "OpenAI" },
|
|
"element_alignment": { "enabled": true, "weight": 1.0, "llm": "Gemini" },
|
|
"product_visibility_general": { "enabled": true, "weight": 1.5, "llm": "OpenAI" }
|
|
}
|
|
}
|
|
```
|
|
|
|
Result: Visible to both Diageo and L'Oreal clients
|
|
|
|
## Web UI Integration
|
|
|
|
### Creating Profiles in Web UI
|
|
|
|
When creating a profile through the web interface, you'll see:
|
|
|
|
1. **Profile Name**: Enter profile name
|
|
2. **Description**: Describe the profile purpose
|
|
3. **Reveal to All Clients**: Checkbox
|
|
- ☑️ **Checked**: Profile visible to all clients
|
|
- ☐ **Unchecked**: Show client selector
|
|
|
|
4. **Client Selector** (if not revealed to all):
|
|
- Select specific clients that can use this profile
|
|
- Multiple clients can be selected
|
|
|
|
### Editing Profiles in Web UI
|
|
|
|
When editing a profile:
|
|
|
|
1. System automatically creates new version
|
|
2. Original profile remains unchanged
|
|
3. Success message shows: "Profile updated to version 2. Original kept as 'profile_name'"
|
|
4. New version becomes active in all client configurations
|
|
|
|
### Profile List View
|
|
|
|
Profiles in the list show:
|
|
- Profile name
|
|
- Version number (if versioned)
|
|
- Visibility status:
|
|
- 🌍 "All Clients" - Global visibility
|
|
- 🔒 "Specific Clients" - Limited visibility
|
|
- Client names listed for limited visibility
|
|
|
|
## Best Practices
|
|
|
|
### Profile Naming
|
|
|
|
- ✅ **Good**: `diageo_premium_2026`, `unilever_sustainability_check`
|
|
- ❌ **Avoid**: `test123`, `my_profile`, `new_new_final_v2`
|
|
|
|
### When to Use Versioning
|
|
|
|
- ✅ Changing check weights
|
|
- ✅ Modifying enabled checks
|
|
- ✅ Updating pass thresholds
|
|
- ✅ Changing profile descriptions
|
|
- ❌ Fixing typos (minor edits can be done manually if needed)
|
|
|
|
### When to Use Client-Specific Visibility
|
|
|
|
- ✅ Brand-specific QC criteria
|
|
- ✅ Confidential checks
|
|
- ✅ Client-requested custom profiles
|
|
- ✅ Experimental profiles for specific clients
|
|
- ❌ General-purpose checks
|
|
- ❌ Standard quality criteria
|
|
|
|
### Version Cleanup
|
|
|
|
Over time, you may accumulate many versions. Consider:
|
|
|
|
1. **Keep Recent Versions**: Keep last 3-5 versions
|
|
2. **Archive Old Versions**: Move to `profiles/archive/` folder
|
|
3. **Document Major Versions**: Add version notes in description
|
|
|
|
```bash
|
|
# Archive old versions (example)
|
|
mkdir -p backend/profiles/archive
|
|
mv backend/profiles/my_profile_v[1-3].json backend/profiles/archive/
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Profile Not Appearing in Client Dropdown
|
|
|
|
Check:
|
|
1. Profile `visibility` setting in JSON file
|
|
2. Client ID in `visible_to_clients` array
|
|
3. `client_config.py` has been updated correctly
|
|
4. Browser cache (refresh page with Ctrl+F5)
|
|
|
|
### Version Number Not Incrementing
|
|
|
|
Check:
|
|
1. Profile was actually modified (not just re-saved)
|
|
2. No errors in browser console
|
|
3. Server logs for any errors
|
|
|
|
### Client Config Not Updating
|
|
|
|
Check:
|
|
1. Write permissions on `client_config.py`
|
|
2. Server logs for update errors
|
|
3. Manual review of `client_config.py` file
|
|
|
|
## Migration Guide
|
|
|
|
### Migrating Existing Profiles
|
|
|
|
Existing profiles without version/visibility fields will:
|
|
- Default to version 1
|
|
- Default to visibility "all"
|
|
- Continue working without changes
|
|
|
|
To add versioning to existing profiles:
|
|
|
|
```bash
|
|
# Add metadata to existing profile
|
|
python -c "
|
|
import json
|
|
profile_path = 'backend/profiles/my_profile.json'
|
|
with open(profile_path, 'r') as f:
|
|
profile = json.load(f)
|
|
profile['version'] = 1
|
|
profile['visibility'] = 'all'
|
|
profile['visible_to_clients'] = []
|
|
profile['created_at'] = '2026-02-02T10:00:00'
|
|
profile['created_by'] = 'system'
|
|
with open(profile_path, 'w') as f:
|
|
json.dump(profile, f, indent=2)
|
|
"
|
|
```
|
|
|
|
## API Reference
|
|
|
|
### Profile Schema
|
|
|
|
```typescript
|
|
interface Profile {
|
|
name: string; // Display name
|
|
description: string; // Profile description
|
|
version: number; // Version number (1, 2, 3, ...)
|
|
visibility: "all" | "client_specific"; // Visibility mode
|
|
visible_to_clients: string[]; // Client IDs (if client_specific)
|
|
created_at: string; // ISO datetime
|
|
created_by: string; // User email
|
|
modified_at?: string; // ISO datetime (if edited)
|
|
modified_by?: string; // User email (if edited)
|
|
previous_version?: string; // Previous version profile_id
|
|
pass_threshold: number; // Pass/fail threshold (0-100)
|
|
weight_scale?: number; // Optional weight scale
|
|
checks: {
|
|
[check_name: string]: {
|
|
enabled: boolean;
|
|
weight: number;
|
|
llm: "OpenAI" | "Gemini";
|
|
description?: string;
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Error Responses
|
|
|
|
```json
|
|
{
|
|
"status": "error",
|
|
"message": "Profile not found"
|
|
}
|
|
```
|
|
|
|
```json
|
|
{
|
|
"status": "error",
|
|
"message": "Profile already exists"
|
|
}
|
|
```
|
|
|
|
```json
|
|
{
|
|
"status": "error",
|
|
"message": "No changes detected"
|
|
}
|
|
```
|
|
|
|
## Support
|
|
|
|
For questions or issues:
|
|
1. Check profile JSON files in `backend/profiles/`
|
|
2. Review `backend/client_config.py` for client assignments
|
|
3. Check server logs for errors
|
|
4. Contact the development team
|