Enhances _set_field_value() to handle empty value structures and
adds detailed logging for debugging CreativeX field issues.
Changes to metadata_extractor_mvp.py:
_set_field_value() Enhancement:
- Handles empty nested dicts by creating value structure
- If field['value']['value'] is empty dict, creates {'value': value}
- If field['value'] is empty dict, creates {'value': {'value': value}}
- Preserves existing behavior for populated structures
CreativeX Score Field Debugging:
- Added score_field_found flag to detect if field exists in mvp_fields
- Logs field structure before attempting to set value
- Shows dict keys to understand nesting
- Catches and logs full traceback on errors
- Errors if CREATIVEX Score field not found in mvp_fields
- Changed log: "from Box" → "from database" (accurate)
CreativeX URL Field:
- Existing logic preserved
- Uses same enhanced _set_field_value()
Purpose:
Diagnose why CreativeX score not appearing in asset representation
even though logs show "Set CREATIVEX Score to: 0"
Next Steps:
Run with --dryrun to see field structure logging and verify values
are being set correctly in the JSON output.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds debugging mode to A2→A3 workflow that builds full asset metadata
but doesn't upload to DAM, displaying complete JSON for field validation.
Changes to A2→A3 Script (a2_to_a3_upload_polling.py):
--dryrun Flag:
- New argument: --dryrun (build metadata but don't upload)
- Displays full asset representation as formatted JSON
- Shows field count
- Shows CreativeX lookup status
- Keeps file in Box (no deletion)
- Logs "DRYRUN MODE" clearly
- Returns success with 'DRYRUN_NO_UPLOAD' as asset_id
Dryrun Output Includes:
- Complete asset_representation JSON (all MVP fields)
- Field count (should be 27 fields)
- CreativeX status (found/missing)
- CreativeX score and URL values
- Clean separation with === lines
Usage:
python scripts/a2_to_a3_upload_polling.py --dryrun
Benefits:
- Debug metadata issues without DAM uploads
- Verify all fields present before going live
- Check CreativeX integration working
- Validate field values and formatting
- Safe testing with production data
Changes to Field Mappings (config/field_mappings.yaml):
Agency Name Fixed:
- Changed: FERRERO.MARKETING.FIELD.AGENCY NAME: "-"
- To: FERRERO.MARKETING.FIELD.AGENCY NAME: "Oliver"
- Exact case as required by DAM
- Comment updated to reflect this is final value
Impact:
- All A2→A3 uploads now have Agency Name = "Oliver"
- Not "Oliver Agency" (wrong)
- Not "-" placeholder (old)
- Exact case: "Oliver" (capital O, lowercase liver)
Use Case:
Run with --dryrun to see full JSON metadata, verify Agency name is
"Oliver", check all 27 MVP fields are present, then remove --dryrun
flag to perform actual uploads.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changes behavior to only log when Box folder is empty, not send emails.
Rationale:
- Empty folder is normal operation (not an error condition)
- Reduces email noise when script runs on cron
- Still logs the event for monitoring
- Similar to other workflows (a1_to_a2, etc.) that don't email when no work to do
Changes:
- Removed notifier.send_email() call for 'creativex_no_files' template
- Enhanced log message: "No PDF files found - this is normal when folder is empty"
- Added: "Script completed successfully with no files to process"
- Still returns success=True (not an error)
Template 'creativex_no_files' retained for potential future use but not called.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces Box metadata template with database lookup for CreativeX scores,
adding automatic fallback to default values when scores are missing.
Changes to A2→A3 Script (a2_to_a3_upload_polling.py):
- Removed: box.get_file_metadata() for CreativeX data
- Added: db.get_creativex_score_by_filename() database lookup
- Uses clean filename (stripped of OMG Job + Tracking ID) for lookup
- Implements fallback when score not found:
- Default score: 0
- Default URL: https://app.creativex.com/preflight/pretests
- Tracks creativex_found flag for email notifications
- Logs warnings when defaults are used
Email Template Updates (notifier.py):
- Shows "✅ CreativeX Score Added: 85 (from database)" when found
- Shows "⚠️ CreativeX Score: Not found - used default (0)" when missing
- Adds orange warning box when defaults used:
- Lists default values (Score: 0, placeholder URL)
- Provides instructions to add score
- References Box folder 350605024645 and scoring script
Benefits:
- Automatic CreativeX lookup (no manual Box metadata entry)
- Graceful degradation (uploads succeed even without scores)
- Clear notification when scores are missing
- Preserves history (uses latest active version)
- No breaking changes (existing workflow continues to work)
Default Value Strategy:
- Score 0 indicates "not scored" but doesn't block upload
- Placeholder URL is valid CreativeX domain
- Email clearly shows when defaults are used
- Provides actionable instructions for adding scores
Workflow Integration:
1. CreativeX PDFs uploaded to folder 350605024645
2. creativex_scoring_storing.py extracts and stores scores
3. A2→A3 automatically looks up scores by filename
4. Uploads proceed with actual scores OR defaults
5. Email indicates which path was taken
Documentation: A2_A3_CREATIVEX_INTEGRATION.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements version counter for re-scored files and cleans up numeric formatting.
Decimal Removal:
- Strip .0 suffix from creativex_id (6864255.0 → 6864255)
- Strip .0 suffix from quality_score (80.0 → 80)
- Converts float → int → string before storing
- Cleaner data for display and DAM integration
Version Tracking:
- Counts total versions per filename (active + superseded)
- Returns version_number in database result
- Logs show version: "Score 80 extracted (Version 3)"
- Email templates display version badges for updates
Email Template Updates:
- Complete template: Shows "Version 3 (Updated)" badge in header
- Includes note: "This is version 3 of this file"
- Partial template: Shows "(Version 3)" inline
- Only displays version info if > 1
Database Changes:
- Query counts ALL versions before insert
- Returns version_number in result dict
- Logs include version in success/update messages
Benefits:
- Clean numeric values without unnecessary decimals
- Users can see if file was re-scored
- Version history visible in emails
- Still preserves all history in database
- A2→A3 integration unaffected (always gets latest active)
Example progression:
Upload 1: Score 80 (no version shown - it's the first)
Upload 2: Score 85 (Version 2 badge shown)
Upload 3: Score 90 (Version 3 badge shown)
Documentation: CREATIVEX_VERSION_UPDATES.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds upsert logic that marks old records as 'superseded' while creating
new 'active' records, preserving full history for audit/analysis.
Changes:
- Updated store_creativex_score() to check for existing filename
- Old records marked status='superseded' before inserting new 'active' record
- Returns is_update flag to indicate if this was an update vs new insert
- Logs score changes (e.g., "Score: 80.0 -> 85.0")
Documentation updates:
- Added "Understanding Status Field" section with soft delete explanation
- Separated queries into "Latest Scores" vs "History/Audit" sections
- Added A2→A3 integration guide with example code
- Documented query logic and behavior table for future integration
- Added migration notes for existing data
Query patterns for A2→A3:
- status='active' → Latest/current score (use this in workflows)
- status='superseded' → Previous scores (history/audit trail)
- get_creativex_score_by_filename() automatically filters for active
Benefits:
- Easy lookup of latest scores (just filter status='active')
- Full history preserved for tracking score changes over time
- No data loss when files are re-scored
- Clear audit trail of when scores changed
Tested and verified:
- Existing record (80.0) marked as superseded
- New record (85.0) created as active
- Queries correctly return only active record
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements new workflow to extract CreativeX quality scores from PDFs
using LlamaExtract AI and store results in PostgreSQL database.
Components added:
- creativex_scoring_storing.py: Main script to process PDFs from Box
- creativex_scores table: Database table with JSONB for full JSON storage
- Database methods: store_creativex_score() and get_creativex_score_by_filename()
- Email templates: creativex_complete, creativex_partial, creativex_no_files
- Configuration: creativex section in config.yaml
- CREATIVEX_DEPLOYMENT.md: Complete deployment and usage guide
Features:
- Monitors Box folder 350605024645 for PDFs
- Extracts scores using LlamaExtract agent "Creativex-Extract"
- Stores 4 key fields (filename, ID, URL, score) + full JSON
- Deletes processed PDFs from Box after successful extraction
- Sends email notifications for success/partial/no-files scenarios
- Manual execution (python scripts/creativex_scoring_storing.py)
Database schema:
- Table: creativex_scores with 10 columns
- Indexes on filename, box_file_id, status for fast lookups
- JSONB column stores complete extraction for future flexibility
Future integration ready:
db.get_creativex_score_by_filename() available for DAM upload workflows
to attach CreativeX metadata during asset processing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added CreativeX fields to MVP fields list:
- FERRERO.TAB.FIELD.CREATIVEX (CreativeX Score from Box)
- FERRERO.FIELD.CREATIVEX LINK (CreativeX URL from Box)
Fixed _update_creativex_fields method:
- Now actually SETS CreativeX Score value (was only logging before)
- Uses _set_field_value() to update the field
- Added try/except for tabular field structure handling
- Both score and URL now properly set from Box metadata
Flow:
1. Box metadata retrieved from Ferrero-DAM-Metadata template
2. creativexScore and creativexUrl extracted
3. Passed to metadata extractor
4. Applied to MVP fields during upload
5. CreativeX data preserved from Box → DAM
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created update_campaign_status.py - targeted campaign status control:
- Update specific campaign by number (--camp C000000078)
- Or search by partial campaign name (--camp "CONTENT SCALING")
- Set to any status (--status A1, A2, A3, etc.)
- Searches all statuses to find the campaign
- If multiple matches, lets user choose
- Interactive yes/no confirmation
- Shows current and target status
- Supports --auth-pfx for mTLS authentication
Usage examples:
python scripts/update_campaign_status.py --camp C000000078 --status A2
python scripts/update_campaign_status.py --camp "KINDER" --status A4
python scripts/update_campaign_status.py --camp C000000551 --status A1 --auth-pfx
Features:
- Required flags: --camp and --status
- Searches across all statuses (A1-A6, B1-B2)
- Partial name matching (case insensitive)
- Multiple match handling with user selection
- Clear success/failure feedback
Updated README with complete documentation and examples
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Created 2 new interactive scripts for manual campaign status control:
1. reset_campaign_to_a1.py
- Resets any campaign from any A# status back to A1
- Allows filtering by specific status (--status A2, A3, A4, etc.)
- Interactive yes/no confirmation for each campaign
- Shows campaign name, number, and current status
- Processes one at a time, waits for user input
- Summary statistics at the end
2. advance_a1_to_a3.py
- Advances A1 campaigns directly to A3 (skips A2)
- Useful for testing A2→A3 upload workflow
- Interactive yes/no confirmation for each campaign
- Shows campaign details before advancing
- Summary statistics at the end
Both scripts:
- Support --auth-pfx flag for mTLS authentication
- Process campaigns one at a time (not batch)
- Wait for user input before proceeding
- Provide clear feedback on success/failure
- Useful for testing and manual workflow control
Updated README.md with complete documentation and examples
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Created config/country_code_mappings.yaml with 165 country mappings
- ISO codes (used in filenames) now map to DAM-specific codes
- Codes under review set to XX as placeholder
- Added load_country_code_mappings() to config_loader.py
- Updated MetadataExtractorMVP to load and apply country mappings
- Added _map_country_code() and _get_field_value() helper methods
- Country mapping applies in both full and folder-only modes
Key mappings:
- BD (Bangladesh) -> BG (DAM code, though appears incorrect)
- DE (Germany) -> DE (same)
- IT (Italy) -> IT (same)
- Most codes under review -> XX (placeholder)
Mapping file can be edited without code changes - updates apply automatically
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major changes:
1. Updated filename_parser.py for new V2 naming convention:
- Spot version now accepts only MST or REF (optional)
- Duration field is now optional
- Tracking ID supports -N suffix for folder-only mode
- Reduced minimum required parts from 9 to 7
- Improved asset type detection logic
2. Added recursive folder scanning to box_client.py:
- New list_folder_files_recursive() method
- Skips first-level job/batch folders
- Preserves folder structure from 2nd level onwards
- Skips hidden folders (starting with . or _)
3. Updated A2→A3 upload workflow:
- Uses recursive folder scanning
- Extracts and logs tracking mode (full vs folder_only)
- Handles subfolder paths for DAM uploads
- Shows folder distribution in logs
4. Added folder-only mode to metadata_extractor_mvp.py:
- New tracking_mode parameter (full/folder_only)
- folder_only mode builds metadata entirely from filename
- New _build_fields_from_filename() method
5. Added DAM subfolder creation to dam_client.py:
- New get_or_create_subfolder_path() method
- Creates matching folder structure in DAM
- Helper methods _find_subfolder_by_name() and _create_folder()
Folder structure behavior:
- Box: DAM-UPLOAD/1234567/Europe/Germany/file.mp4
- DAM: 01. Final Assets/Europe/Germany/file.mp4
- Job folder (1234567) is skipped, structure preserved from 2nd level
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Changed from single-campaign mode to process all A4 campaigns in one run
- Added summary statistics (total found, webhooks sent, already processed, failed)
- Since webhooks are lightweight operations, processing all at once is efficient
- Duplicate prevention still works via database tracking
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Changed campaign.get('campaign_id', 'N/A') to use 'or' operator
- Now returns 'UNKNOWN' instead of None when campaign_id is missing
- Prevents NULL constraint violation in campaign_status table
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed database.py line 479: Changed 'CURRENT_TIMESTAMP' string to actual datetime
- Added datetime import for proper UTC timestamp generation
- This fixes the PostgreSQL error: invalid input syntax for type timestamp
- Added migration file for campaign_status table
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces 20+ second job polling with instant folder search.
PERFORMANCE IMPROVEMENT:
Before: Poll job 10 times, 2 sec each = 20+ seconds per upload
After: No polling, instant return + optional folder search
CHANGES:
1. Removed job polling from upload_asset()
- No more 20 second waits
- Returns job_id immediately for async uploads
- Adds 'is_async' flag to response
2. NEW METHOD: find_asset_by_filename_in_folder()
- Fast search by filename in folder
- Can be called after batch uploads complete
- Returns actual asset ID instantly
UPLOAD FLOW:
Immediate Response (201):
→ Returns asset ID immediately
→ Log: "Upload successful (immediate): file.jpg → Asset ID: abc123"
Async Response (202):
→ Returns job ID immediately (no waiting!)
→ Log: "Upload accepted (async): file.jpg → Job ID: job456"
→ Log: "Note: Job processing in background. Asset ID can be found later."
FINDING ASSET ID LATER (OPTIONAL):
After batch uploads, call once per folder:
```python
# Upload all files first (fast!)
for file in files:
result = dam.upload_asset(...)
job_ids.append(result['job_id'])
# Then search folder for actual IDs (one API call)
for filename in filenames:
asset_id = dam.find_asset_by_filename_in_folder(folder_id, filename)
```
BENEFITS:
✓ No 20 second waits per file
✓ Batch uploads can run quickly
✓ Optional post-upload search for asset IDs
✓ Single API call to get all IDs
USE CASES:
- Fast uploads: Don't need immediate asset ID
- Batch processing: Upload many files quickly
- Later retrieval: Search folder when needed
- Status updates: Can update campaign without waiting
The job_id is stored and can be used for tracking.
Actual asset_id can be retrieved later if needed.
Changes:
- scripts/shared/dam_client.py
- Removed polling from upload_asset()
- Added find_asset_by_filename_in_folder() method
- Returns immediately with job_id for async
- Added 'is_async' flag to response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Allows keeping uploaded files in Box for testing/debugging purposes.
NEW FEATURE: --keep-files Flag
- Optional flag for testing/debugging
- Prevents deletion of files from Box after upload
- Files remain in Box folder after successful DAM upload
USAGE:
Default (delete files after upload):
python scripts/a2_to_a3_upload_polling.py
Keep files in Box (testing):
python scripts/a2_to_a3_upload_polling.py --keep-files
Combined with A3update:
python scripts/a2_to_a3_upload_polling.py --keep-files --A3update
HOW IT WORKS:
1. Upload file to DAM (always happens)
2. Store in database (always happens)
3. If --keep-files flag:
- Skip Box file deletion
- Log: "--keep-files flag set - File kept in Box: filename.jpg"
4. If no flag (default):
- Delete file from Box
- Log: "Deleted file from Box: filename.jpg"
LOGGING:
```
With flag:
--keep-files flag set - File kept in Box: my_file.jpg
Without flag:
Deleted file from Box: my_file.jpg
```
USE CASES:
- Testing: Upload multiple times without re-uploading to Box
- Debugging: Keep files to inspect Box metadata
- Development: Test upload logic without losing files
- Backup: Maintain Box copies during initial testing
PRODUCTION NOTE:
For production, don't use this flag - files should be deleted
after successful upload to avoid duplicates on next run.
BOTH FLAGS TOGETHER:
python scripts/a2_to_a3_upload_polling.py --keep-files --A3update
- Uploads file to DAM
- Keeps file in Box
- Updates campaign A2→A3
- Perfect for end-to-end testing
Changes:
- scripts/a2_to_a3_upload_polling.py
- Added --keep-files flag
- Added keep_files parameter to process_box_file()
- Conditional Box file deletion
- Enhanced logging for both modes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Allows manual campaign status update A2→A3 after file upload.
NEW FEATURE: --A3update Flag
- Optional flag for testing purposes
- Forces campaign status update A2→A3 after file upload
- Extracts campaign ID from uploaded asset's metadata
- Updates campaign status in DAM
USAGE:
Default (no status update):
python scripts/a2_to_a3_upload_polling.py
With status update (testing):
python scripts/a2_to_a3_upload_polling.py --A3update
HOW IT WORKS:
1. Upload file to DAM (always happens)
2. If --A3update flag set:
- Extract campaign ID from master asset metadata
- Look in inherited_metadata_collections for campaign container
- Update campaign status A2 → A3
- Log success/failure
LOGGING:
```
--A3update flag set - Attempting to update campaign status
Found campaign ID: abc123def456
Updating campaign status A2 → A3...
✓ Campaign status updated successfully: A2 → A3
```
USE CASES:
- Testing: Quickly update campaign status after single upload
- Manual workflow: Force status update without waiting for all assets
- Development: Test status update functionality
PRODUCTION NOTE:
For production, typically don't use this flag.
Campaign should stay at A2 until ALL localized assets uploaded.
Changes:
- scripts/a2_to_a3_upload_polling.py
- Added argparse import
- Added --A3update flag
- Added campaign status update logic
- Extracts campaign ID from full_metadata
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhanced upload process with better logging and true asset ID retrieval.
UPLOAD LOGGING IMPROVEMENTS:
- Now logs asset representation being sent:
- Model ID
- Security policy count
- Metadata field count
- Helps debug upload issues
ASSET ID vs JOB ID FIX:
Previously: When DAM returned job_handle, we stored job_id as asset_id
Now: Poll the job to get the ACTUAL asset ID
NEW METHOD: _poll_job_for_asset_id()
- Polls /v6/jobs/{job_id} endpoint (max 10 attempts, 2 sec delay)
- Checks multiple response locations for asset_id
- Logs job status progress
- Returns actual asset ID when job completes
- Falls back to job_id if asset ID not found
RESPONSE HANDLING:
1. If 'asset_resource_list' in response:
- Direct asset ID (synchronous upload)
- Log: "Upload successful: file.jpg → Asset ID: abc123"
2. If 'job_handle' in response:
- Async job (needs polling)
- Log: "Upload accepted (async): file.jpg → Job ID: job123"
- Log: "Polling job for actual asset ID..."
- Poll job status every 2 seconds
- Log: "Job status (attempt X): running/completed"
- Log: "✓ Job completed → Asset ID: abc123"
BENEFITS:
✓ True asset ID stored in database (not job ID)
✓ Better upload debugging with detailed logs
✓ Can track job progress
✓ Handles both sync and async uploads correctly
LOGGING EXAMPLE:
```
Uploading: my_file.jpg
Parent Folder ID: abc123
Asset Representation:
Model ID: ferrero.model.video
Security Policies: 2
Metadata Fields: 27
Upload accepted (async): my_file.jpg → Job ID: job456
Polling job for actual asset ID...
Job status (attempt 1): running
Job status (attempt 2): completed
✓ Job completed → Asset ID: asset789
```
Changes:
- scripts/shared/dam_client.py
- Added upload logging before API call
- Added _poll_job_for_asset_id() method
- Updated upload_asset() to poll jobs for asset ID
- Returns both asset_id and job_id in result
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major README overhaul with complete deployment and configuration guide.
Moved old docs to tests/ folder for archive.
README.md UPDATES (880 lines - completely rewritten):
✓ Table of contents with navigation
✓ Complete overview of all 4 workflows + daily report
✓ Detailed authentication section (OAuth2 vs mTLS)
✓ Box-config.json location explanation
✓ Server deployment step-by-step guide
✓ Database setup (Docker + native PostgreSQL)
✓ Cron job examples for all workflows
✓ Comprehensive troubleshooting section
✓ Security checklist
✓ Monitoring and log rotation details
✓ Common SQL queries
✓ File structure diagram
KEY SECTIONS ADDED:
1. What's Included - All 5 scripts explained
2. Quick Start - Local setup guide
3. Server Deployment - 6-step process with commands
4. Workflows - Detailed process for each (A1→A2, A5→A6, B1→B2, A2→A3, Daily Report)
5. Authentication - OAuth2 vs mTLS with examples
6. Configuration - All .env variables documented
7. Database - Schema, setup, queries
8. Monitoring - Logs, emails, database queries
9. Troubleshooting - Common issues + solutions
10. File Structure - Complete directory tree
BOX-CONFIG.JSON LOCATION DOCUMENTED:
✓ Must be one folder up from Python-Version
✓ Referenced as ../Box-config.json in config.yaml
✓ Server deployment instructions include copying both files
✓ Troubleshooting section explains file not found errors
MTLS DOCUMENTATION:
✓ Different base URL explained (dev-auth.app-api.ferrero.com)
✓ --auth-pfx flag usage
✓ Whitelisted IP requirement noted
✓ Certificate testing commands
REORGANIZATION:
- Moved old DEPLOYMENT.md → tests/DEPLOYMENT.md (archive)
- Moved old WORKFLOW_DIAGRAMS.md → tests/WORKFLOW_DIAGRAMS.md (archive)
- New DEPLOYMENT_GUIDE.md is the current deployment doc
- README.md is now comprehensive one-stop documentation
Changes:
- Python-Version/README.md (completely rewritten, 880 lines)
- Moved 2 old docs to tests/ folder
- Added test files to tests/ folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Critical fix: mTLS uses completely different API endpoint than OAuth2.
KEY CHANGE:
OAuth2 and mTLS now use different base URLs automatically based on auth method.
CONFIGURATION:
- OAuth2: https://ppr.dam.ferrero.com/otmmapi
- mTLS: https://dev-auth.app-api.ferrero.com/00003/mm
URLs are automatically selected based on --auth-pfx flag:
- No flag: Uses DAM_BASE_URL (OAuth2 endpoint)
- --auth-pfx: Uses DAM_MTLS_BASE_URL (mTLS endpoint)
IMPLEMENTATION:
1. .env: Added DAM_MTLS_BASE_URL variable
2. config.yaml: Added mtls_base_url configuration
3. dam_client.py: Auto-selects base_url in __init__ based on use_mtls flag
4. All API calls automatically use correct endpoint
EXAMPLE ENDPOINT TRANSFORMATION:
OAuth2: https://ppr.dam.ferrero.com/otmmapi/v6/search/text
mTLS: https://dev-auth.app-api.ferrero.com/00003/mm/v6/search/text
(Same path, different host/prefix)
TESTING STATUS:
✓ Certificate loads successfully
✓ Correct base URL selected based on mode
⚠️ HTTP 403 from current IP (likely IP whitelist)
✓ Ready to test from whitelisted IP location
ALL SCRIPTS UPDATED:
✓ a1_to_a2_download.py - Uses correct URL with --auth-pfx
✓ a5_to_a6_download.py - Uses correct URL with --auth-pfx
✓ b1_to_b2_download.py - Uses correct URL with --auth-pfx
✓ test_connection.py - Uses correct URL with --auth-pfx
NEW DEBUG SCRIPT:
- test_mtls_debug.py - Detailed request/response logging
BACKWARD COMPATIBILITY:
✓ OAuth2 completely unchanged (default)
✓ No impact on existing workflows
✓ Can test mTLS from whitelisted IP when ready
Next: Test from whitelisted IP location to verify mTLS works end-to-end.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds session management for mTLS to handle "No session exists" errors.
UPDATES:
- Added session storage in __init__ for mTLS mode
- Updated _make_api_request to use requests.Session with mTLS
- Session persists certificate and cookies across requests
- Added OTDSTicket cookie handling
CURRENT STATUS:
✓ Certificate loads successfully
✓ Connection test passes
⚠️ Search campaigns returns HTTP 401 "No session exists"
This suggests mTLS may need:
1. Different API endpoints than OAuth2
2. Additional session initialization step
3. Specific headers or authentication flow
4. Contact DAM API team for mTLS documentation
OAuth2 remains default and fully functional.
Use --auth-pfx flag to test mTLS when ready.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Ensures all workflows send email notifications even when no assets are found.
NEW EMAIL TEMPLATES:
1. a1_to_a2_no_assets - Warns when A1 campaign has no master assets
2. b1_to_b2_no_assets - Warns when B1 campaign has no global master assets
Both templates use orange warning theme and explain:
- Campaign is set to A1/B1 but no assets found
- Status NOT updated (remains at current status)
- Script will retry on next run
- Suggests verifying assets exist in folder
Changes:
- a1_to_a2_download.py: Send email when total_assets == 0
- b1_to_b2_download.py: Send email when total_assets == 0
- notifier.py: Add 2 new warning templates
This completes email coverage for all scenarios:
✓ Success (all assets processed)
✓ Partial (some failed)
✓ No assets found (campaign empty)
✓ No rejections (A5→A6 specific)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Critical fix and UX improvements for all workflow email notifications.
CRITICAL FIX:
- A5→A6 now correctly searches Final Assets folder (is_global=True)
- Previously searched Master Assets folder (wrong location)
- Now finds NOT APPROVED rework assets correctly
TESTED SUCCESSFULLY:
✓ Found 6 total assets in Final Assets folder
✓ Filtered 4 NOT APPROVED assets correctly
✓ Skipped 2 folders without ECOMMERCE STATUS field
✓ Downloaded and uploaded 4 assets to Box Revisions folder
✓ Email sent with rejection details
✓ Status updated A5→A6
EMAIL TEMPLATE STYLING UNIFICATION:
All templates now use consistent modern styling matching a5_to_a6_rejections:
- Colored header bars with centered titles
- Bordered info boxes with left accent bars
- Card-based asset display with colored headers
- Consistent spacing and typography
- Professional color scheme
Templates Updated:
1. a1_to_a2_complete - Green theme (#28a745)
2. a1_to_a2_partial - Orange theme (#ff9800)
3. a2_to_a3_complete - Green theme (#28a745)
4. a2_to_a3_file_uploaded - Green/Blue theme
5. b1_to_b2_complete - Blue theme (#1976d2)
6. b1_to_b2_partial - Orange theme (#ff9800)
7. upload_failed - Red theme (#d32f2f)
All templates keep existing data/functionality, only styling improved.
Color Scheme:
- Success: Green (#28a745)
- Warning/Partial: Orange (#ff9800)
- Error: Red (#d32f2f)
- Info: Blue (#1976d2)
- Highlights: Yellow (#ffc107)
Changes:
- Python-Version/scripts/a5_to_a6_download.py (is_global=True fix)
- Python-Version/scripts/shared/notifier.py (7 templates restyled)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major enhancements to all workflow scripts with recursive search and detailed rejection tracking.
NEW FEATURES:
1. Recursive Folder Search (ALL workflows: A1→A2, A5→A6, B1→B2)
- Searches subfolders within Master/Final Assets folders
- Preserves folder structure in Box
- Adds 'folder_path' attribute to each asset
2. NOT APPROVED Filtering (A5→A6 ONLY)
- Only downloads assets with ECOMMERCE STATUS = "NOT APPROVED"
- Skips approved/other status assets
- Logs rejected vs skipped counts
3. Rejection Details Extraction (A5→A6)
- Extracts comments from 3 reviewers: Approver, Legal, IA&CC
- Includes certifier names and dates
- Displays in detailed email notifications
CHANGES BY FILE:
dam_client.py:
- NEW: _get_assets_recursive() - Recursively searches folders
- UPDATED: get_master_assets() - Now uses recursive search, adds folder_path to assets
- NEW: is_asset_not_approved() - Checks FERRERO.FIELD.ECOMMERCE STATUS
- NEW: extract_rejection_details() - Extracts all rejection comments from 10 fields
box_client.py:
- UPDATED: upload_with_tracking_id() - Added subfolder_path parameter
- NEW: _get_or_create_subfolder_path() - Creates/navigates Box subfolders
- Preserves DAM folder structure in Box uploads
a1_to_a2_download.py:
- Added folder_path extraction from assets
- Pass subfolder_path to Box upload
- Logs subfolder info during processing
b1_to_b2_download.py:
- Added folder_path extraction from assets
- Pass subfolder_path to Box upload
- Logs subfolder info during processing
a5_to_a6_download.py:
- Filter assets for NOT APPROVED status ONLY
- Extract rejection details for each asset
- Pass subfolder_path to Box upload
- Updated email data with rejection_details
- Handle "no rejections" scenario with email
- Updated logging to show rejected vs skipped counts
notifier.py:
- REPLACED: a5_to_a6_complete → a5_to_a6_rejections
- Detailed HTML template with rejection sections
- Shows Approver, Legal, and IA&CC rejections
- Styled with red warnings and bordered sections
- NEW: a5_to_a6_no_rejections template
- Green success message when no rejected assets found
- UPDATED: a5_to_a6_partial - Now uses rejected_assets
FIELD IDs EXTRACTED (A5→A6):
- FERRERO.FIELD.ECOMMERCE STATUS (primary check)
- FERRERO.MARKETING.FIELD.CERTIFIER COMMENT
- FERRERO.FIELD.ECOMMERCE CERTIFIER
- FERRERO.MARKETING.FIELD.APPROVAL DATE
- FERRERO.MARKETING.FIELD.LEGAL COMMENT
- FERRERO.FIELD.LEGAL CERTIFER (typo in field ID)
- FERRERO.MARKETING.FIELD.LEGAL APPROVAL DATE
- FERRERO.MARKETING.FIELD.IA CC COMMENT
- FERRERO.MARKETING.FIELD.IA CERTIFIER
- FERRERO.MARKETING.FIELD.IA CC APPROVAL DATE
TESTING:
✓ All connections working (DAM, Box, Database)
✓ A5→A6 script executes correctly
✓ Recursive search working
✓ NOT APPROVED filtering working
✓ "No rejections" email sent successfully
✓ Folder structure preserved in logs
WORKFLOW IMPACTS:
- A1→A2: Now searches recursively, preserves folder structure
- A5→A6: Filters for NOT APPROVED only, shows rejection details
- B1→B2: Now searches recursively, preserves folder structure
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Database Schema:
✅ Added local_campaign_id VARCHAR(50) column
✅ Stores the immediate campaign the asset belongs to (C000000551)
Enhanced Extraction:
✅ extract_global_campaign_reference() now returns 3 values:
- global_master_campaign_id (C000000068)
- global_master_folder_id (676f2bcde4c7...)
- local_campaign_id (C000000551)
✅ Extracts FERRERO.FIELD.CAMPAIGN ID from same collection
✅ Only sets local_campaign_id if that collection has global reference
✅ Logs all three IDs when found
A1→A2 Script:
✅ Passes local_campaign_id to store_master_asset()
✅ Stores complete campaign relationship
Database Now Stores:
- tracking_id: ABC123 (unique 6-char)
- opentext_id: 0008a50... (DAM asset ID)
- local_campaign_id: C000000551 (immediate campaign)
- global_master_campaign_id: C000000068 (global master)
- global_master_folder_id: 676f2bcde4c7... (global folder)
Example Relationship:
- Asset downloaded from Local Campaign C000000551
- That campaign references Global Master C000000068
- All three IDs stored for complete traceability
Database schema now complete with full campaign relationship tracking!
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Database Schema:
✅ Added global_master_campaign_id VARCHAR(50) column
✅ Added global_master_folder_id VARCHAR(255) column
✅ Stores relationship between local campaigns and their global masters
Database Module:
✅ Added extract_global_campaign_reference() method
✅ Searches inherited_metadata_collections for L7+ - CAMPAIGN containers
✅ Extracts FERRERO.FIELD.GLOBAL CAMPAIGN REFERENCE field
✅ Extracts container_id as global_master_folder_id
✅ Returns dict with both IDs
A1→A2 Script:
✅ Calls db.extract_global_campaign_reference() for each asset
✅ Passes global_master_campaign_id to store_master_asset()
✅ Passes global_master_folder_id to store_master_asset()
✅ Logs when Global Campaign Reference found
Example Data Stored:
- Local Campaign C000000551 asset
- global_master_campaign_id: C000000068
- global_master_folder_id: 676f2bcde4c7bcf7ef783e97f7495069bf50b6bc
Usage:
This data enables tracking which Global Master a local asset came from.
Can query all local assets for a specific Global Master campaign.
Foundation for future cross-campaign features.
Based on EXTRACTION_GUIDE.md implementation pattern.
Note: B1→B2 workflow NOT updated (those ARE the global masters)
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Issue: Folder named 'MASTERS_NUTELLA PLANT-BASED LAUNCH-'
Missing: Campaign number (C000000068)
Fix: Include campaign number in folder name
- campaign_id: MASTERS_C000000068
- campaign_name: NUTELLA_PLANT_BASED_LAUNCH
- Result: MASTERS_C000000068-NUTELLA_PLANT_BASED_LAUNCH
Format:
MASTERS_[CampaignNumber]-[CampaignName]
Examples:
- MASTERS_C000000068-NUTELLA_PLANT_BASED_LAUNCH
- MASTERS_C000000069-KINDER_SURPRISE_GLOBAL
Spaces and dashes in campaign name converted to underscores.
🤖 Generated with Claude Code
Issue: Folder named 'MASTERS_NUTELLA PLANT-BASED LAUNCH-' (with trailing dash)
Cause: Passing entire folder name as campaign_id, empty campaign_name
Fix: Use BoxClient's built-in naming logic
- campaign_id: 'MASTERS' (prefix)
- campaign_name: 'NUTELLA_PLANT_BASED_LAUNCH' (cleaned)
- Result: MASTERS-NUTELLA_PLANT_BASED_LAUNCH
Changes:
- Replace spaces with underscores
- Replace dashes with underscores
- BoxClient adds dash between ID and name
- Final format: MASTERS-Campaign_Name
Example Results:
Before: MASTERS_NUTELLA PLANT-BASED LAUNCH-
After: MASTERS-NUTELLA_PLANT_BASED_LAUNCH
Clean, consistent folder naming for Global Masters!
🤖 Generated with Claude Code
Box Metadata Extraction Fixed:
Template Name:
❌ Was: 'Ferrero-DAM-Metadata' (with dashes and caps)
✅ Now: 'ferrerodammetadata' (lowercase, no dashes)
Field Names:
❌ Was: 'CreativeX Score', 'CreativeX URL'
✅ Now: 'creativexScore', 'creativexUrl' (camelCase)
Test Results with File 2035459900168:
✅ Template found successfully
✅ creativexScore: 90
✅ creativexUrl: https://www.bbc.com✅ Metadata extraction working!
A2→A3 workflow will now:
1. Download file from Box
2. Read ferrerodammetadata template
3. Extract creativexScore and creativexUrl
4. Update CREATIVEX fields in asset representation
5. Upload to DAM with CreativeX data
Test: Upload a file to Box with ferrerodammetadata template applied
and the A2→A3 script will extract and use those values!
🤖 Generated with Claude Code
Implementation Complete:
✅ Box metadata extraction method
✅ Retrieves from 'Ferrero-DAM-Metadata' template
✅ Extracts 'CreativeX Score' and 'CreativeX URL'
✅ Updates CREATIVEX LINK field in asset representation
✅ Integrated into A2→A3 workflow
Testing Note:
File 2035459900168 does not have metadata template applied yet.
To Test:
1. In Box, select a file in folder 348526703108
2. Right-click → More Actions → Metadata
3. Apply template: Ferrero-DAM-Metadata
4. Fill in:
- CreativeX Score: 85
- CreativeX URL: https://creativex.com/report/12345
5. Run A2→A3 script
6. Check logs for: 'CreativeX URL from Box: ...'
7. Verify CreativeX fields in uploaded DAM asset
Implementation ready - awaiting file with metadata template for testing.
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Major Feature: Box Metadata Integration
box_client.py:
✅ Added get_file_metadata() method
✅ Reads 'Ferrero-DAM-Metadata' template from Box files
✅ Extracts 'CreativeX Score' and 'CreativeX URL' fields
✅ Returns dict with score and url
a2_to_a3_upload_polling.py:
✅ Calls box.get_file_metadata() before download
✅ Logs Box metadata retrieved
✅ Passes box_metadata to build_mvp_asset_representation()
metadata_extractor_mvp.py:
✅ Added box_metadata parameter to build_mvp_asset_representation()
✅ Added _update_creativex_fields() method
✅ Updates FERRERO.FIELD.CREATIVEX LINK with URL from Box
✅ Logs CreativeX Score (tabular field - needs special handling)
Flow:
1. File uploaded to Box by agency
2. Agency adds metadata using Ferrero-DAM-Metadata template
3. Script reads CreativeX Score and URL from Box metadata
4. Updates MVP fields with Box metadata values
5. Uploads to DAM with CreativeX data
Field Mapping:
- Box: 'CreativeX URL' → DAM: FERRERO.FIELD.CREATIVEX LINK
- Box: 'CreativeX Score' → DAM: FERRERO.TAB.FIELD.CREATIVEX (logged, needs structure)
Next: Test with file that has Box metadata template applied
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Email Templates Added:
1. b1_to_b2_complete
- Subject: Global Master Assets Downloaded
- Shows campaign name, ID, asset count
- Lists all processed assets with tracking IDs and Box URLs
- Notes Box folder: 349261192115
- Status updated: B1 → B2
2. b1_to_b2_partial
- Subject: Partial Download - Global Campaign
- Shows successful and failed assets separately
- Each asset listed with name, tracking ID, error
- Notes status NOT updated (remains B1)
- Mentions automatic retry
Webhook Removed from B1→B2:
- B1→B2 workflow now only sends email (no webhook)
- Webhook only for A1→A2 workflow
- Simplified B1→B2 notifications
Email will now render properly with formatted HTML instead of raw dict.
Next B1→B2 run will send properly formatted email!
🤖 Generated with Claude Code
Changed: 'All assets processed - Updating status A1 → A2'
To: 'All assets processed - Updating status B1 → B2'
All log messages now correctly reference B1→B2 workflow.
Test output should now show:
- Searching for B1 Global campaigns
- All assets processed - Updating status B1 → B2
- ✓ Status updated: B1 → B2
Complete and correct! 🎊🤖 Generated with Claude Code
Fixes:
1. PHP: Fixed function name
- Changed findFinalAssetsFolder() → findUploadFolder()
- This function already looks for Final Assets folder
- Now PHP interface works without fatal error
2. Python: Search for Global comm campaigns
- Added campaign_type parameter to search_campaigns()
- B1→B2 uses: campaign_type='Global comm'
- A1→A2 uses: campaign_type='Local Adaptation' (default)
3. Python: Fixed log messages
- 'Searching for B1 Global campaigns' (not A1)
- 'No B1 campaigns found' (not A1)
4. Box Folder Configuration
- B1→B2 uses folder: 349261192115
- Folder naming: MASTERS_Campaign_Name
B1→B2 Now:
✅ Searches Global comm campaigns
✅ Filters for B1 status
✅ Uses Final Assets folder (05. not 01.)
✅ Uploads to correct Box folder (349261192115)
✅ Names folders: MASTERS_NUTELLA_PLANT-BASED_LAUNCH
Test:
1. Refresh PHP app - should load now
2. B1→B2 tab should work
3. Python script should find B1 campaigns
🤖 Generated with Claude Code
Created complete B1→B2 automation script:
✅ Based on tested a1_to_a2_download.py structure
✅ Searches for campaigns with status B1
✅ Searches Global comm campaigns (not Local Adaptation)
✅ Downloads Global Master assets from DAM
✅ Uploads to Box with tracking IDs (folder: 348304357505)
✅ Stores in PostgreSQL with full metadata
✅ Updates status B1 → B2 when all successful
✅ Sends webhook with B1→B2 status change
✅ Sends email notifications (b1_to_b2_complete, b1_to_b2_partial)
✅ Log rotation (28 files, 10MB each)
✅ Single-run mode (process one campaign and exit)
Usage:
cd Python-Version
source venv/bin/activate
python scripts/b1_to_b2_download.py
Cron Setup:
*/5 * * * * python scripts/b1_to_b2_download.py
Test Campaign Available:
- NUTELLA PLANT-BASED LAUNCH
- Folder ID: 676f2bcde4c7bcf7ef783e97f7495069bf50b6bc
- Status: B1
Complete B1→B2 automation ready for testing!
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Issue: CREATIVEX fields still not appearing
Root Cause: FERRERO.FIELD.CREATIX is a CATEGORY, not a field within a category
Fix:
- Check category ID/name for 'CREATIX' or 'CreativeX'
- When CREATIVEX category found, extract ALL items within it
- Handle both tables and direct fields in CREATIVEX category
- Show fields even if empty (displays structure)
Structure:
Category: FERRERO.FIELD.CREATIX (name: CreativeX)
├─ Table: FERRERO.TABULAR.FIELD.CREATIVEX (Confidence)
│ └─ Field: FERRERO.TAB.FIELD.CREATIVEX (Platform > Rating %)
└─ Field: FERRERO.FIELD.CREATIVEX LINK (CreativeX Hyperlink)
Test Results:
✅ Extracted 2 CREATIVEX fields
✅ Platform > Rating (%): (empty)
✅ CreativeX Hyperlink: (empty)
Now purple CREATIVEX section will appear in metadata viewer!
🤖 Generated with Claude Code
A2→A3 Email Now Includes:
File Details:
✅ Original filename (from Box with Job# and Tracking ID)
✅ Clean filename (stripped for DAM)
✅ DAM Asset ID
✅ Tracking ID
Processing Details:
✅ Master Asset ID (source asset)
✅ Upload folder ID (where it went in DAM)
✅ Box folder ID (where it came from)
Complete Step-by-Step:
✅ Downloaded from Box (folder 348526703108)
✅ Loaded master metadata from database
✅ Built 27 MVP fields
✅ Updated Description from filename
✅ Updated Language from filename
✅ Set State to Local
✅ Stripped Job# and Tracking ID
✅ Uploaded to DAM
✅ Deleted from Box
Now both A1→A2 and A2→A3 emails are extremely verbose!
🤖 Generated with Claude Code
Email Template Enhancements:
1. A1→A2 Complete Email - Now Shows:
✅ Campaign name and number
✅ Asset count
✅ List of ALL processed assets with:
- Asset name
- Tracking ID
- Box file ID
- Box URL (clickable link)
2. A1→A2 Partial Email - Now Shows:
✅ Campaign details
✅ Total/successful/failed counts
✅ List of SUCCESSFUL assets with:
- Asset name
- Tracking ID
- Box URL
✅ List of FAILED assets with:
- Asset name
- Specific error message
✅ Note about automatic retry
3. A2→A3 File Uploaded Email - Shows:
✅ Original filename (with Job# and Tracking ID)
✅ Clean filename (stripped)
✅ DAM Asset ID
✅ Tracking ID
✅ Note that file was deleted from Box
Benefits:
- Know exactly which assets succeeded/failed
- Can click Box URLs to verify files
- Can track specific errors per asset
- Don't need to check logs for details
- Full visibility into automation status
Example Partial Email:
━━━━━━━━━━━━━━━━━━━━━━━
Campaign Partially Processed
Campaign: KSURPRISE LOCAL (C000000123)
Total: 3 | Successful: 1 | Failed: 2
✅ Successfully Processed (1):
• asset1.mp4 (Tracking ID: ABC123)
Box URL: https://app.box.com/file/123❌ Failed Assets (2):
• asset2.mp4 (Error: Network timeout)
• asset3.mp4 (Error: Invalid metadata)
━━━━━━━━━━━━━━━━━━━━━━━
Also Updated DEPLOYMENT.md:
- Added Key Features section
- Documented log rotation (28 files, 10MB each)
- Documented Box file deletion
- Documented per-file email notifications
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
