- Enhanced _get_assets_recursive() to properly identify folders vs files
- Added resource_type field validation (more reliable than asset_type)
- Created whitelist of 40+ known file extensions for accurate detection
- Fixes HTTP 404 errors when processing folders like '01. REFERENCE FILES'
- Applies to both A1->A2 and B1->B2 workflows
- Added ARTESIA.FIELD.ASSET_ID to MVP fields in field_mappings.yaml
- Updated metadata_extractor_mvp.py to accept master_opentext_id parameter
- Added _add_master_asset_id_field() and _get_field_id() helper methods
- Modified a2_to_a3_upload_polling.py to pass master asset's opentext_id
- Field is populated with original master asset's DAM ID for derivative tracking
- Field is omitted for new assets (tracking ID with -N suffix)
- Covers both A2→A3 standard derivatives and A5→A6 reworked assets
- Modified _set_field_value to include 'type': 'string' in all code paths
- Adds type field when updating existing CreativeX URL field
- Ensures consistent structure whether creating or updating field
- Added 'type': 'string' to FERRERO.FIELD.CREATIVEX LINK value structure
- Fixes DAM validation error for CreativeX URL field
- Structure now matches DAM requirements
- Updated creativex_scoring_storing.py to map multiple placements to platforms
- Modified get_mapped_platform to get_mapped_platforms (returns list)
- Updated a2_to_a3_upload_polling.py to retrieve platforms list from DB
- Enhanced metadata_extractor_mvp.py to build multi-value CreativeX field
- Added DAM-CX mappings.csv for channel/placement to platform mapping
- Supports single channel with multiple placements generating multiple Platform^Score values
Enables asset type to be updated from derivative filename and refactors
_update_fields() to use filename_updates configuration dynamically.
Field Mappings Configuration (field_mappings.yaml):
Added to filename_updates:
- FERRERO.FIELD.MKTG.ASSET TYPE:
source: asset_type
required: true
Now updates from derivative filename:
- ROC_TEST-E2E2_EHI_1x1_DE_de.png → Asset Type = "EHI"
Metadata Extractor Refactor (metadata_extractor_mvp.py):
Old _update_fields():
- Hardcoded field updates (ASSET NAME, DESCRIPTION, STATE)
- Not using filename_updates configuration
- Required code changes to add new fields
New _update_fields():
- Dynamically processes filename_updates from config
- Supports transform: uppercase/lowercase
- Supports any source field from parsed_filename
- Uses forced_values from config (was hardcoded before)
- Add new fields via config, no code changes needed
Configuration-Driven Updates:
- ARTESIA.FIELD.ASSET NAME ← clean_filename
- ARTESIA.FIELD.ASSET DESCRIPTION ← subject_title
- FERRERO.FIELD.MKTG.ASSET TYPE ← asset_type (NEW)
- MAIN_LANGUAGES ← language_code (uppercase)
- FERRERO.FIELD.STATE ← "Local" (forced value)
Benefits:
- Asset type now correctly populated from filename
- Configuration-driven (add fields without code changes)
- Cleaner code (uses config instead of hardcoded logic)
- Forced values also configurable
- Easier to maintain and extend
Example:
Filename: ROC_TEST-E2E2_EHI_1x1_DE_de.png
Parsed asset_type: "EHI"
Field FERRERO.FIELD.MKTG.ASSET TYPE updated to: "EHI"
Impact:
All A2→A3 uploads will now have correct Asset Type from derivative
filename instead of inheriting from master (which may be different).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Stores master asset CreativeX scores from DAM metadata during A1→A2
download for reference/reporting purposes (not used in uploads).
Database Changes:
creativex_scores table:
- Added: tracking_id VARCHAR(6) column
- Added: idx_creativex_tracking_id index
- Updated comment: status can be 'active', 'superseded', or 'master-cx-score'
Status Values:
- 'active' - Current derivative score (from PDF extraction)
- 'superseded' - Old derivative score (version history)
- 'master-cx-score' - Master asset score (from A1→A2 DAM metadata) ← NEW
Migration SQL (for existing databases):
ALTER TABLE creativex_scores ADD COLUMN tracking_id VARCHAR(6);
CREATE INDEX idx_creativex_tracking_id ON creativex_scores(tracking_id);
Database Method Updates (database.py):
store_creativex_score() signature:
- Added: tracking_id parameter (optional, default None)
- Added: status parameter (optional, default 'active')
Logic:
- If status='master-cx-score': Simple insert, no versioning
- If status='active': Soft delete versioning as before
- Always stores tracking_id if provided
A1→A2 Script Updates (a1_to_a2_download.py):
New Function: extract_creativex_from_dam_metadata()
- Searches metadata_element_list for CREATIVEX fields
- Extracts FERRERO.TAB.FIELD.CREATIVEX (score)
- Extracts FERRERO.FIELD.CREATIVEX LINK (url)
- Returns dict with score/url or None if not found
- Handles tabular field structure for score
- Handles nested value structure for URL
Integration:
- After successful master asset storage
- Extracts CreativeX from asset metadata
- If found: Stores in creativex_scores with status='master-cx-score'
- Links to master via tracking_id
- Logs when score found/stored
- Logs "normal" when not found (not all masters are scored)
Use Cases:
A2→A3 Upload:
- Still uses filename-based lookup ONLY ✅
- No changes to A2→A3 logic ✅
- Master scores not used for uploads ✅
Reporting/Analytics Tools:
- Can query master score by tracking_id
- Compare master vs derivative scores
- Track score improvements
- Audit trail
Query Examples:
-- Get master score for tracking ID
SELECT * FROM creativex_scores
WHERE tracking_id = '7xXgKp' AND status = 'master-cx-score';
-- Get derivative score for filename
SELECT * FROM creativex_scores
WHERE filename = 'file.mp4' AND status = 'active';
Test Record Created:
- Filename: nutella_pbased.jpg
- Tracking ID: 7xXgKp
- Score: 85
- Status: master-cx-score
Benefits:
- Historical reference of master scores
- Enables score comparison analytics
- No impact on A2→A3 upload logic
- Automatic extraction during A1→A2
- Optional (works even if masters don't have scores)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Complete rewrite of filename parser to support new field order where
Subject/Asset moved up and Country/Language moved down, plus new
Social Media field.
BREAKING CHANGE: V2.1 Structure (November 2025)
Old (V1):
[JOB]_[BRAND]_[COUNTRY]_[LANG]_[SUBJECT]_[ASSET]_[SPOT]_[DUR]_[RATIO]_[TRACKING]
New (V2.1):
[JOB]_[BRAND]_[SUBJECT]_[ASSET]_[DUR]_[RATIO]_[SPOT]_[COUNTRY]_[LANG]_[SOCIAL]_[TRACKING]
Field Position Changes:
- Subject Title: Position 5 → 3 (MOVED UP)
- Asset Type: Position 6 → 4 (MOVED UP)
- Duration: Position 8 → 5 (MOVED UP)
- Aspect Ratio: Position 9 → 6 (MOVED UP)
- Spot Version: 7 → 7 (SAME)
- Country Code: Position 3 → 8 (MOVED DOWN)
- Language Code: Position 4 → 9 (MOVED DOWN)
- Social Media: NEW → Position 10
- Tracking ID: Position 10 → 11
New Social Media Field:
- Field: social_media_version
- Position: 10 (after language, before tracking)
- Format: 3 uppercase letters
- Codes: FBP, FBR, IGF, IGR (expandable)
- Optional: Only present for social media assets
Parse Algorithm Changes:
- Positions 1-4 now: Job, Brand, Subject, Asset (fixed)
- Positions 5-11: Pattern-based detection (flexible)
- Duration detected by \d+S pattern
- Aspect ratio detected by \d+x\d+ or contains 'x'
- Spot detected by MST/REF
- Country detected by 2 upper alpha (after ratio)
- Language detected by 2-3 lower alpha (after country)
- Social detected by known codes (after language)
- Tracking ID detected by 6 alphanumeric + optional -N
strip_upload_components() Updated:
Now outputs: [BRAND]_[SUBJECT]_[ASSET]_[DUR]_[RATIO]_[SPOT]_[COUNTRY]_[LANG]_[SOCIAL]
- Includes social media version if present
- Still strips job number and tracking ID
Testing:
All 7 test cases from specification passed:
✅ All fields present
✅ Minimal (no duration/social/tracking)
✅ No duration
✅ No spot version
✅ With -N tracking (folder-only mode)
✅ No social media (most common)
✅ No tracking ID
Example:
Input: 1234567_RAF_TEST_OLV_6S_1x1_REF_GL_it_IGF_abc123.mp4
Parsed: brand=RAF, subject=TEST, country=GL, lang=it, social=IGF
Clean: RAF_TEST_OLV_6S_1x1_REF_GL_it_IGF.mp4
Backward Compatibility:
None - system not live yet, clean cutover to V2.1 format only.
Backup: filename_parser_v1_backup.py contains old version for reference.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes issue where CreativeX score field was not appearing in final upload
because it didn't exist in the master metadata from DAM.
Problem:
- Master metadata from A1→A2 doesn't include CREATIVEX fields (new fields)
- _update_creativex_fields() only UPDATED existing fields
- If field not present, it logged error but didn't add the field
- Result: CREATIVEX score missing from upload, only URL appeared
Solution:
- Check if CREATIVEX Score field exists in mvp_fields
- If NOT found: Create and append field with proper structure
- If found: Update value as before
- Same logic for CREATIVEX URL field
Field Structures Created:
CREATIVEX Score (FERRERO.TAB.FIELD.CREATIVEX):
- Type: MetadataTableField (tabular field)
- Parent: FERRERO.TABULAR.FIELD.PLATFORMRATING
- Data type: INTEGER
- Value structure: {'value': {'value': score}}
CREATIVEX URL (FERRERO.FIELD.CREATIVEX LINK):
- Type: MetadataField (regular field)
- Data type: CHAR
- Value structure: {'value': {'value': url}}
Logging:
- Changed from ERROR to WARNING when field not found
- Logs "adding it now" instead of just error
- Confirms field added with value
Impact:
Both CreativeX fields will now appear in uploads even if master
metadata doesn't have them (common for older campaigns downloaded
before CreativeX integration).
Testing:
Run with --dryrun to verify both CREATIVEX fields in JSON output.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds detailed logging to trace exactly how field values are being set
and diagnose why CreativeX score/URL aren't appearing in final JSON.
Changes to _set_field_value():
- Logs field ID being updated
- Logs current field['value'] structure BEFORE setting
- Logs which code path is taken (nested vs created)
- Logs field['value'] structure AFTER setting
- Shows full JSON structure at each step
Output Example:
_set_field_value called for: FERRERO.TAB.FIELD.CREATIVEX with value: 85
Current field['value']: {
"is_locked": false,
"domain_value": false,
...
}
Created field['value'] = {'value': {'value': 85}}
After setting, field['value']: {
"value": {
"value": 85
}
}
Purpose:
Diagnose why CreativeX fields show empty value dicts in asset
representation even though logs say "Set CREATIVEX Score to: 0".
This verbose logging will show:
1. What the field structure looks like before we set it
2. Which code path is executed
3. What the field structure looks like after we set it
4. Whether the value is actually being placed in the right location
Run with --dryrun to see full debug output without uploading.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Corrects misleading log messages and adds debugging for URL field.
Changes:
- Changed: "Updating CreativeX URL from Box"
- To: "Updating CreativeX URL from database"
- Added url_field_found flag
- Added URL field structure logging
- Added error handling with traceback
- Logs error if URL field not found in mvp_fields
Now both CreativeX fields log correctly:
- "Updating CreativeX Score from database: 0"
- "Updating CreativeX URL from database: https://..."
Accurate logging shows data source is PostgreSQL creativex_scores
table, not Box metadata templates (which are no longer used).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhances _set_field_value() to handle empty value structures and
adds detailed logging for debugging CreativeX field issues.
Changes to metadata_extractor_mvp.py:
_set_field_value() Enhancement:
- Handles empty nested dicts by creating value structure
- If field['value']['value'] is empty dict, creates {'value': value}
- If field['value'] is empty dict, creates {'value': {'value': value}}
- Preserves existing behavior for populated structures
CreativeX Score Field Debugging:
- Added score_field_found flag to detect if field exists in mvp_fields
- Logs field structure before attempting to set value
- Shows dict keys to understand nesting
- Catches and logs full traceback on errors
- Errors if CREATIVEX Score field not found in mvp_fields
- Changed log: "from Box" → "from database" (accurate)
CreativeX URL Field:
- Existing logic preserved
- Uses same enhanced _set_field_value()
Purpose:
Diagnose why CreativeX score not appearing in asset representation
even though logs show "Set CREATIVEX Score to: 0"
Next Steps:
Run with --dryrun to see field structure logging and verify values
are being set correctly in the JSON output.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces Box metadata template with database lookup for CreativeX scores,
adding automatic fallback to default values when scores are missing.
Changes to A2→A3 Script (a2_to_a3_upload_polling.py):
- Removed: box.get_file_metadata() for CreativeX data
- Added: db.get_creativex_score_by_filename() database lookup
- Uses clean filename (stripped of OMG Job + Tracking ID) for lookup
- Implements fallback when score not found:
- Default score: 0
- Default URL: https://app.creativex.com/preflight/pretests
- Tracks creativex_found flag for email notifications
- Logs warnings when defaults are used
Email Template Updates (notifier.py):
- Shows "✅ CreativeX Score Added: 85 (from database)" when found
- Shows "⚠️ CreativeX Score: Not found - used default (0)" when missing
- Adds orange warning box when defaults used:
- Lists default values (Score: 0, placeholder URL)
- Provides instructions to add score
- References Box folder 350605024645 and scoring script
Benefits:
- Automatic CreativeX lookup (no manual Box metadata entry)
- Graceful degradation (uploads succeed even without scores)
- Clear notification when scores are missing
- Preserves history (uses latest active version)
- No breaking changes (existing workflow continues to work)
Default Value Strategy:
- Score 0 indicates "not scored" but doesn't block upload
- Placeholder URL is valid CreativeX domain
- Email clearly shows when defaults are used
- Provides actionable instructions for adding scores
Workflow Integration:
1. CreativeX PDFs uploaded to folder 350605024645
2. creativex_scoring_storing.py extracts and stores scores
3. A2→A3 automatically looks up scores by filename
4. Uploads proceed with actual scores OR defaults
5. Email indicates which path was taken
Documentation: A2_A3_CREATIVEX_INTEGRATION.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements version counter for re-scored files and cleans up numeric formatting.
Decimal Removal:
- Strip .0 suffix from creativex_id (6864255.0 → 6864255)
- Strip .0 suffix from quality_score (80.0 → 80)
- Converts float → int → string before storing
- Cleaner data for display and DAM integration
Version Tracking:
- Counts total versions per filename (active + superseded)
- Returns version_number in database result
- Logs show version: "Score 80 extracted (Version 3)"
- Email templates display version badges for updates
Email Template Updates:
- Complete template: Shows "Version 3 (Updated)" badge in header
- Includes note: "This is version 3 of this file"
- Partial template: Shows "(Version 3)" inline
- Only displays version info if > 1
Database Changes:
- Query counts ALL versions before insert
- Returns version_number in result dict
- Logs include version in success/update messages
Benefits:
- Clean numeric values without unnecessary decimals
- Users can see if file was re-scored
- Version history visible in emails
- Still preserves all history in database
- A2→A3 integration unaffected (always gets latest active)
Example progression:
Upload 1: Score 80 (no version shown - it's the first)
Upload 2: Score 85 (Version 2 badge shown)
Upload 3: Score 90 (Version 3 badge shown)
Documentation: CREATIVEX_VERSION_UPDATES.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds upsert logic that marks old records as 'superseded' while creating
new 'active' records, preserving full history for audit/analysis.
Changes:
- Updated store_creativex_score() to check for existing filename
- Old records marked status='superseded' before inserting new 'active' record
- Returns is_update flag to indicate if this was an update vs new insert
- Logs score changes (e.g., "Score: 80.0 -> 85.0")
Documentation updates:
- Added "Understanding Status Field" section with soft delete explanation
- Separated queries into "Latest Scores" vs "History/Audit" sections
- Added A2→A3 integration guide with example code
- Documented query logic and behavior table for future integration
- Added migration notes for existing data
Query patterns for A2→A3:
- status='active' → Latest/current score (use this in workflows)
- status='superseded' → Previous scores (history/audit trail)
- get_creativex_score_by_filename() automatically filters for active
Benefits:
- Easy lookup of latest scores (just filter status='active')
- Full history preserved for tracking score changes over time
- No data loss when files are re-scored
- Clear audit trail of when scores changed
Tested and verified:
- Existing record (80.0) marked as superseded
- New record (85.0) created as active
- Queries correctly return only active record
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements new workflow to extract CreativeX quality scores from PDFs
using LlamaExtract AI and store results in PostgreSQL database.
Components added:
- creativex_scoring_storing.py: Main script to process PDFs from Box
- creativex_scores table: Database table with JSONB for full JSON storage
- Database methods: store_creativex_score() and get_creativex_score_by_filename()
- Email templates: creativex_complete, creativex_partial, creativex_no_files
- Configuration: creativex section in config.yaml
- CREATIVEX_DEPLOYMENT.md: Complete deployment and usage guide
Features:
- Monitors Box folder 350605024645 for PDFs
- Extracts scores using LlamaExtract agent "Creativex-Extract"
- Stores 4 key fields (filename, ID, URL, score) + full JSON
- Deletes processed PDFs from Box after successful extraction
- Sends email notifications for success/partial/no-files scenarios
- Manual execution (python scripts/creativex_scoring_storing.py)
Database schema:
- Table: creativex_scores with 10 columns
- Indexes on filename, box_file_id, status for fast lookups
- JSONB column stores complete extraction for future flexibility
Future integration ready:
db.get_creativex_score_by_filename() available for DAM upload workflows
to attach CreativeX metadata during asset processing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added CreativeX fields to MVP fields list:
- FERRERO.TAB.FIELD.CREATIVEX (CreativeX Score from Box)
- FERRERO.FIELD.CREATIVEX LINK (CreativeX URL from Box)
Fixed _update_creativex_fields method:
- Now actually SETS CreativeX Score value (was only logging before)
- Uses _set_field_value() to update the field
- Added try/except for tabular field structure handling
- Both score and URL now properly set from Box metadata
Flow:
1. Box metadata retrieved from Ferrero-DAM-Metadata template
2. creativexScore and creativexUrl extracted
3. Passed to metadata extractor
4. Applied to MVP fields during upload
5. CreativeX data preserved from Box → DAM
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Created config/country_code_mappings.yaml with 165 country mappings
- ISO codes (used in filenames) now map to DAM-specific codes
- Codes under review set to XX as placeholder
- Added load_country_code_mappings() to config_loader.py
- Updated MetadataExtractorMVP to load and apply country mappings
- Added _map_country_code() and _get_field_value() helper methods
- Country mapping applies in both full and folder-only modes
Key mappings:
- BD (Bangladesh) -> BG (DAM code, though appears incorrect)
- DE (Germany) -> DE (same)
- IT (Italy) -> IT (same)
- Most codes under review -> XX (placeholder)
Mapping file can be edited without code changes - updates apply automatically
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major changes:
1. Updated filename_parser.py for new V2 naming convention:
- Spot version now accepts only MST or REF (optional)
- Duration field is now optional
- Tracking ID supports -N suffix for folder-only mode
- Reduced minimum required parts from 9 to 7
- Improved asset type detection logic
2. Added recursive folder scanning to box_client.py:
- New list_folder_files_recursive() method
- Skips first-level job/batch folders
- Preserves folder structure from 2nd level onwards
- Skips hidden folders (starting with . or _)
3. Updated A2→A3 upload workflow:
- Uses recursive folder scanning
- Extracts and logs tracking mode (full vs folder_only)
- Handles subfolder paths for DAM uploads
- Shows folder distribution in logs
4. Added folder-only mode to metadata_extractor_mvp.py:
- New tracking_mode parameter (full/folder_only)
- folder_only mode builds metadata entirely from filename
- New _build_fields_from_filename() method
5. Added DAM subfolder creation to dam_client.py:
- New get_or_create_subfolder_path() method
- Creates matching folder structure in DAM
- Helper methods _find_subfolder_by_name() and _create_folder()
Folder structure behavior:
- Box: DAM-UPLOAD/1234567/Europe/Germany/file.mp4
- DAM: 01. Final Assets/Europe/Germany/file.mp4
- Job folder (1234567) is skipped, structure preserved from 2nd level
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed database.py line 479: Changed 'CURRENT_TIMESTAMP' string to actual datetime
- Added datetime import for proper UTC timestamp generation
- This fixes the PostgreSQL error: invalid input syntax for type timestamp
- Added migration file for campaign_status table
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces 20+ second job polling with instant folder search.
PERFORMANCE IMPROVEMENT:
Before: Poll job 10 times, 2 sec each = 20+ seconds per upload
After: No polling, instant return + optional folder search
CHANGES:
1. Removed job polling from upload_asset()
- No more 20 second waits
- Returns job_id immediately for async uploads
- Adds 'is_async' flag to response
2. NEW METHOD: find_asset_by_filename_in_folder()
- Fast search by filename in folder
- Can be called after batch uploads complete
- Returns actual asset ID instantly
UPLOAD FLOW:
Immediate Response (201):
→ Returns asset ID immediately
→ Log: "Upload successful (immediate): file.jpg → Asset ID: abc123"
Async Response (202):
→ Returns job ID immediately (no waiting!)
→ Log: "Upload accepted (async): file.jpg → Job ID: job456"
→ Log: "Note: Job processing in background. Asset ID can be found later."
FINDING ASSET ID LATER (OPTIONAL):
After batch uploads, call once per folder:
```python
# Upload all files first (fast!)
for file in files:
result = dam.upload_asset(...)
job_ids.append(result['job_id'])
# Then search folder for actual IDs (one API call)
for filename in filenames:
asset_id = dam.find_asset_by_filename_in_folder(folder_id, filename)
```
BENEFITS:
✓ No 20 second waits per file
✓ Batch uploads can run quickly
✓ Optional post-upload search for asset IDs
✓ Single API call to get all IDs
USE CASES:
- Fast uploads: Don't need immediate asset ID
- Batch processing: Upload many files quickly
- Later retrieval: Search folder when needed
- Status updates: Can update campaign without waiting
The job_id is stored and can be used for tracking.
Actual asset_id can be retrieved later if needed.
Changes:
- scripts/shared/dam_client.py
- Removed polling from upload_asset()
- Added find_asset_by_filename_in_folder() method
- Returns immediately with job_id for async
- Added 'is_async' flag to response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhanced upload process with better logging and true asset ID retrieval.
UPLOAD LOGGING IMPROVEMENTS:
- Now logs asset representation being sent:
- Model ID
- Security policy count
- Metadata field count
- Helps debug upload issues
ASSET ID vs JOB ID FIX:
Previously: When DAM returned job_handle, we stored job_id as asset_id
Now: Poll the job to get the ACTUAL asset ID
NEW METHOD: _poll_job_for_asset_id()
- Polls /v6/jobs/{job_id} endpoint (max 10 attempts, 2 sec delay)
- Checks multiple response locations for asset_id
- Logs job status progress
- Returns actual asset ID when job completes
- Falls back to job_id if asset ID not found
RESPONSE HANDLING:
1. If 'asset_resource_list' in response:
- Direct asset ID (synchronous upload)
- Log: "Upload successful: file.jpg → Asset ID: abc123"
2. If 'job_handle' in response:
- Async job (needs polling)
- Log: "Upload accepted (async): file.jpg → Job ID: job123"
- Log: "Polling job for actual asset ID..."
- Poll job status every 2 seconds
- Log: "Job status (attempt X): running/completed"
- Log: "✓ Job completed → Asset ID: abc123"
BENEFITS:
✓ True asset ID stored in database (not job ID)
✓ Better upload debugging with detailed logs
✓ Can track job progress
✓ Handles both sync and async uploads correctly
LOGGING EXAMPLE:
```
Uploading: my_file.jpg
Parent Folder ID: abc123
Asset Representation:
Model ID: ferrero.model.video
Security Policies: 2
Metadata Fields: 27
Upload accepted (async): my_file.jpg → Job ID: job456
Polling job for actual asset ID...
Job status (attempt 1): running
Job status (attempt 2): completed
✓ Job completed → Asset ID: asset789
```
Changes:
- scripts/shared/dam_client.py
- Added upload logging before API call
- Added _poll_job_for_asset_id() method
- Updated upload_asset() to poll jobs for asset ID
- Returns both asset_id and job_id in result
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Major README overhaul with complete deployment and configuration guide.
Moved old docs to tests/ folder for archive.
README.md UPDATES (880 lines - completely rewritten):
✓ Table of contents with navigation
✓ Complete overview of all 4 workflows + daily report
✓ Detailed authentication section (OAuth2 vs mTLS)
✓ Box-config.json location explanation
✓ Server deployment step-by-step guide
✓ Database setup (Docker + native PostgreSQL)
✓ Cron job examples for all workflows
✓ Comprehensive troubleshooting section
✓ Security checklist
✓ Monitoring and log rotation details
✓ Common SQL queries
✓ File structure diagram
KEY SECTIONS ADDED:
1. What's Included - All 5 scripts explained
2. Quick Start - Local setup guide
3. Server Deployment - 6-step process with commands
4. Workflows - Detailed process for each (A1→A2, A5→A6, B1→B2, A2→A3, Daily Report)
5. Authentication - OAuth2 vs mTLS with examples
6. Configuration - All .env variables documented
7. Database - Schema, setup, queries
8. Monitoring - Logs, emails, database queries
9. Troubleshooting - Common issues + solutions
10. File Structure - Complete directory tree
BOX-CONFIG.JSON LOCATION DOCUMENTED:
✓ Must be one folder up from Python-Version
✓ Referenced as ../Box-config.json in config.yaml
✓ Server deployment instructions include copying both files
✓ Troubleshooting section explains file not found errors
MTLS DOCUMENTATION:
✓ Different base URL explained (dev-auth.app-api.ferrero.com)
✓ --auth-pfx flag usage
✓ Whitelisted IP requirement noted
✓ Certificate testing commands
REORGANIZATION:
- Moved old DEPLOYMENT.md → tests/DEPLOYMENT.md (archive)
- Moved old WORKFLOW_DIAGRAMS.md → tests/WORKFLOW_DIAGRAMS.md (archive)
- New DEPLOYMENT_GUIDE.md is the current deployment doc
- README.md is now comprehensive one-stop documentation
Changes:
- Python-Version/README.md (completely rewritten, 880 lines)
- Moved 2 old docs to tests/ folder
- Added test files to tests/ folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Critical fix: mTLS uses completely different API endpoint than OAuth2.
KEY CHANGE:
OAuth2 and mTLS now use different base URLs automatically based on auth method.
CONFIGURATION:
- OAuth2: https://ppr.dam.ferrero.com/otmmapi
- mTLS: https://dev-auth.app-api.ferrero.com/00003/mm
URLs are automatically selected based on --auth-pfx flag:
- No flag: Uses DAM_BASE_URL (OAuth2 endpoint)
- --auth-pfx: Uses DAM_MTLS_BASE_URL (mTLS endpoint)
IMPLEMENTATION:
1. .env: Added DAM_MTLS_BASE_URL variable
2. config.yaml: Added mtls_base_url configuration
3. dam_client.py: Auto-selects base_url in __init__ based on use_mtls flag
4. All API calls automatically use correct endpoint
EXAMPLE ENDPOINT TRANSFORMATION:
OAuth2: https://ppr.dam.ferrero.com/otmmapi/v6/search/text
mTLS: https://dev-auth.app-api.ferrero.com/00003/mm/v6/search/text
(Same path, different host/prefix)
TESTING STATUS:
✓ Certificate loads successfully
✓ Correct base URL selected based on mode
⚠️ HTTP 403 from current IP (likely IP whitelist)
✓ Ready to test from whitelisted IP location
ALL SCRIPTS UPDATED:
✓ a1_to_a2_download.py - Uses correct URL with --auth-pfx
✓ a5_to_a6_download.py - Uses correct URL with --auth-pfx
✓ b1_to_b2_download.py - Uses correct URL with --auth-pfx
✓ test_connection.py - Uses correct URL with --auth-pfx
NEW DEBUG SCRIPT:
- test_mtls_debug.py - Detailed request/response logging
BACKWARD COMPATIBILITY:
✓ OAuth2 completely unchanged (default)
✓ No impact on existing workflows
✓ Can test mTLS from whitelisted IP when ready
Next: Test from whitelisted IP location to verify mTLS works end-to-end.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
