Wires B-series (global) campaigns into OMG using the same Box
automation as A-series. Mirrors the A1/A4 lifecycle for B1/B4.
- b1_to_b2_download: after B2 status update, mark live=YES status=B2
and upload live_campaigns_global_<ts>.csv to the existing Box folder
(BOX_LIVE_CAMPAIGNS_FOLDER_ID, 352181382858 in PROD). Filename keeps
the live_campaigns_ prefix so the existing OMG automation rule picks
it up.
- b4_box_uploader (new): polls DAM for status B4, marks live=NO, regens
the global CSV. Mirrors a4_box_uploader.
- a4_box_uploader: reads prior status before overwriting; if it was
B-series, regenerate the global CSV instead. b4_box_uploader does the
symmetric A-series fallback. Defensive in case DAM doesn't enforce
type-specific status transitions.
- database: add get_all_live_global_campaigns() (status LIKE 'B%').
Tighten get_all_live_campaigns() to status LIKE 'A%' so any cross-type
rows can't leak into the wrong CSV.
- orchestrator + orchestrator-prod: register B4 Box Uploader at 10min.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- LTD DAM code confirmed by client: licensingtranslationdocument (was placeholder)
- EOL + LTD: IP Rights forced to "No" (was "Yes")
- EOL + LTD: Remove CreativeX URL and score (not applicable to legal asset types)
- EOL: Description forced to "Legal Studio Name"
- Reorder _apply_asset_type_overrides() to run after _update_creativex_fields()
so overrides have true final precedence (Box CreativeX was clobbering removals)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DAM stores the CreativeX tabular cell as '<platform>^<score>', e.g.
'DV360^100'. Add format_cx_score_for_display() and apply at the point
where the email asset dict is built — both new-download and skipped
paths. Raw value stays in creativex_scores.quality_score so all platform
info is preserved for queries; only the email display is reshaped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diagnostic confirmed FERRERO.TAB.FIELD.CREATIVEX (score) lives at depth 2
in B1 master metadata — nested under FERRERO.TABULAR.FIELD.CREATIVEX
inside a category — and FERRERO.FIELD.CREATIVEX LINK lives at depth 1.
The flat top-level walk used previously never reached them, so live B1
runs and the backfill both reported zero CX scores. Updated extractor
in b1_to_b2_download.py and the inline copy in
backfill_b1_creativex_scores.py to descend recursively.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous version only looked at top-level metadata_element_list, which
contains categories — actual fields nest under each category. Now
recursively descends through all nested metadata_element_list arrays
and counts every element_id at any depth, then searches the full set
for CX/score/quality hints. Reports max nesting depth and the depth at
which each CX-flavored ID was found.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psycopg2 performs %-substitution when params are passed to execute(),
so 'M%' in the LIKE clause was being interpreted as a positional
placeholder, raising IndexError when there's only one real %s (LIMIT).
Escape as 'M%%' so it's preserved as a literal percent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read-only script that samples B1 global masters from master_assets and
reports: top-level keys in full_metadata, presence of
metadata.metadata_element_list, and any element_ids matching
creativex/cx/score/quality (case-insensitive). Helps diagnose why the CX
backfill found 0 matches — distinguishes "client masters have no CX
score yet" from "CX field uses a different element_id than A1".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Walks master_assets for B1 global masters (tracking_id LIKE 'M%' AND
local_campaign_id IS NULL), extracts CreativeX score from full_metadata
JSONB, and inserts into creativex_scores with status='b1-master-cx-score'.
Idempotent — relies on the existing tracking_id dedup in
db.store_creativex_score, so re-runs are safe. Supports --dry-run for
preview before applying.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracts CreativeX score and URL from DAM master metadata during the
B1→B2 download, persists to creativex_scores with new status
'b1-master-cx-score' (dedup by tracking_id), and surfaces the score in
the b1_to_b2_complete and b1_to_b2_partial emails — falling back to
"No CreativeX Score" when the master has no score yet. Skipped
already-downloaded assets backfill from full_metadata JSONB on next pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds SDA as a new asset type for License claim translations supporting
the EOL (External Legal Opinion) workflow.
- SDA maps to externallegalopinion in DAM (same as EOL).
- Field overrides match EOL (Agency = "-", Prod Company = "-",
Languages = Global, IP Right = Yes, Licensing = No, validity dates
removed) plus a fixed Description: "Translation of License claim -
For approval purposes only".
- Added asset_type_overrides section to field_mappings_ppr.yaml; it
was missing, so EOL overrides weren't actually applying on PPR.
Both EOL and SDA blocks are now defined for both PPR and PROD.
- _apply_asset_type_overrides now appends a simple string field when
the override targets a field not yet in mvp_fields, so the SDA
description is set even if the filename has no subject_title.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Campaign managers often create the campaign in DAM before assets are
uploaded, so an empty Master Assets folder is the normal pre-asset state
rather than a failure. Stop marking these as permanently failed and stop
emailing on every poll.
- increment_a1_retry() gains mark_failed_at_max param; empty-folder path
passes False so the campaign keeps polling indefinitely until assets
appear (or the DAM status changes).
- Empty-folder branch now skips silently on every poll and sends a single
warning email at poll 20 (~1 hour at the 3-min cadence) so genuinely
stuck campaigns still surface.
- New a1_to_a2_no_assets_warning email template — one-time soft warning,
no permanent-failure language.
- Existing reset_a1_retry() on successful A1→A2 still clears the counter
when assets eventually appear.
- Other folder-error paths (folder not found, etc.) keep the original
3-retry-then-fail behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The A1→A2 uploader processes up to 2 campaigns per run. Permanently-failed
campaigns were skipped only inside the loop, so they still consumed slots
and could starve the rest of the queue indefinitely. Filter them out
before the slice so eligible campaigns get processed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a campaign is re-opened (status reset to A1/B1 after new files are
added), the tool correctly skips already-downloaded assets but the email
report and CSV previously listed the whole folder as "processed", which
was misleading. Reports now show "Total: 14 (12 previously downloaded,
2 new this run)" with new assets in full detail and previously-downloaded
assets in a compact list. B1→B2 CSV gains a Status column matching A1→A2.
DAM subfolder "WND_PCS 2026 2.0" was being treated as a file because
".0" was not in the known extensions list and defaulted to is_folder=False.
This caused an HTTP 404 on download since it's a folder, not a file.
Added numeric-only extension check (.0, .1, etc.) to the folder detection
logic so the script correctly recurses into versioned subfolders and
downloads the assets inside them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DAM subfolder "WND_PCS 2026 2.0" was being treated as a downloadable
asset because ".0" passed the existing extension check. Added safeguard
to skip items with numeric-only extensions (e.g. .0, .1) which are
version numbers in folder names, not real files.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously the script re-downloaded and re-uploaded all assets on every
retry, even those already successfully stored in DB and Box. For large
campaigns (1300+ assets) this caused unnecessary load and duplicate uploads.
Now checks DB via find_global_master_by_opentext_id() before downloading.
Assets already in DB with a valid Box URL are skipped and counted toward
the processed total, so only genuinely failed assets are retried.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds EOL as a new asset type with field overrides for both PPR and PROD:
- Asset type maps to 'externallegalopinion' in DAM
- Agency Name = "-", Production House = "-"
- Main Languages = "Global"
- IP Rights = "Yes", Licensing = "No"
- Validity dates removed
Also adds VOD platform code and removes OLV asset type.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Folder-only mode (-N suffix files) was sending minimal metadata that DAM
rejected with "unmarshalling parameter" error. Now uses a reference
asset_representation_template.json as the base for all metadata fields,
ensuring the full field structure (column_name, data_type, domain_id, etc.)
the DAM API requires. Also fixes default/forced value handling to use
DomainValue format for domained fields from the template.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Folder-only mode (-N suffix files) was sending simplified metadata that
PROD DAM rejected with "unmarshalling parameter" error. Updated to use
DomainValue format for domained fields, correct asset type field ID
(FERRERO.FIELD.MKTG.ASSET TYPE), asset type code mapping (e.g. SND→sound),
validity dates, and forced values from config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A1→A2 now looks up the opentext_id in master_assets for an M-prefixed record
from B1→B2 and stores it as global_master_tracking_id on the local asset record.
This provides traceability from local campaign assets back to their global master
without changing any existing workflow logic or DAM metadata.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously, re-scored assets with a DAM timestamp suffix (e.g. _2026-03-13-05-53-36)
were treated as new files, leaving multiple 'active' records. Now strips the timestamp
and uses LIKE matching so all variants of the same base asset are properly superseded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mailgun silently drops emails with multiple recipients in the to field.
Send individual API calls per recipient and split comma-separated addresses.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mailgun API is used when MAILGUN_API_KEY and MAILGUN_DOMAIN are set,
with SMTP as fallback for PPR. Also fixes A2→A3 batch subject line
that was rendering Jinja2 syntax literally instead of substituting values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Was generating FERRERO.TABULAR.FIELD.MARKETING_TAG (underscore) but DAM
expects FERRERO.TABULAR.FIELD.MARKETING.TAG (dot). Added explicit mapping
for tabular field parent table IDs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VIDEO_POST_PROD_COMPANY and AUDIO_POST_PROD_COMPANY are not domain fields
but were being wrapped with DomainValue, causing unmarshalling errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The _build_fields_from_filename method was using {"value": "..."} without
the required {"type": "string", "value": "..."} structure, causing
unmarshalling errors on the DAM API for -N suffix uploads.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes unmarshalling error on DAM upload when using -N suffix files. The API
requires the DomainValue structure when domain_value is true.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The filename_updates logic was only updating field['value'] (singular) but for
tabular fields like MAIN_LANGUAGES, the DAM reads from field['values'] (plural
array). This caused the master's original language (e.g. "Global") to persist
instead of the correct language from the filename (e.g. "PL").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When no CreativeX score is found for a file, the system was sending a
generic placeholder URL (app.creativex.com/preflight/pretests) to the DAM.
Now sends no URL at all, so only files with actual CreativeX scores get a URL.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove PPR-only gates so PROD supports the same MASTERASSETIDS tabular
field and multi-master ID parsing as PPR. DAM deployment scheduled for
Feb 18 — do not push until then.
Changes:
- filename_parser: Remove is_ppr check, allow multi-master ID parsing in PROD
- a2_to_a3: Populate master_opentext_ids for single-master PROD case
- dam_client: Remove PPR-only skip on domain registration
- metadata_extractor_mvp: Update docstrings only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The new CreativeX API format stores channel/publisher at the top level
of full_extraction_data instead of inside a data.ferrero_mapped_platforms
wrapper. Add fallback mapping so platforms are correctly populated for
DAM uploads.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube Ads was missing from the DAM-CX mappings CSV, causing empty
Platform > Rating fields for YouTube assets. Also adds a fallback that
derives the CreativeX platform from the filename social media code (e.g.
YTA -> YouTube) when the database has no mapped platforms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Populates master_opentext_ids for single-master case so uploads use the
tabular FERRERO.MASTERASSETIDS field instead of the ARTESIA.FIELD.ASSET_ID
fallback. Reverts the workaround from 6517a4f now that the field is being
configured in PROD DAM.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CreativeX lookup now falls back to tracking ID search when filename match fails
(handles mismatched naming from CreativeX PDFs). strip_upload_components now
only removes job number and tracking ID, keeping social media codes (YTA, DV3,
etc.) in the clean filename. Updated SOCIAL_MEDIA_CODES from 4 to 39 codes
sourced from the Ferrero naming tool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PROD DAM rejects FERRERO.MASTERASSETIDS as it only exists in PPR. Remove the
single-master-to-list conversion so PROD uses the existing single-ID field
(master_opentext_id) instead. Will be re-added when client configures the
tabular field in PROD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The notifier variable was referenced inside process_box_file but never passed
as a parameter, causing NameError for any file hitting the Master Tracking ID
check. Also changed the check from case-insensitive (.upper().startswith('M'))
to case-sensitive (.startswith('M')) to avoid false positives on random tracking
IDs like mviSv5.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The PROD a2_to_a3 script referenced master_opentext_ids without defining it,
causing NameError for all file uploads. Brings in multi-master tracking ID
support from PPR: filename parser handles multiple IDs (PPR) or single ID
(PROD), metadata extractor supports MASTERASSETIDS tabular field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A1→A2 now handles re-processing when campaign is reset to A1 after adding new
master assets. Existing assets reuse tracking IDs and skip Box upload, new assets
are processed normally. Also includes PPR domain registration for multiple master
asset IDs in a2_to_a3 and dam_client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace exposed database credentials and SQL commands in A1 permanently failed notification email with support contact information (optical@oliver.agency).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem:
- Retry logic only triggered for empty folders (total_assets == 0)
- When "Master Assets" folder doesn't exist, error thrown BEFORE retry check
- Exception caught by outer try/except, sent old upload_failed template
- No database tracking, emails sent every 3 minutes indefinitely
Solution:
- Added retry logic to outer exception handler
- Detects folder/assets errors and applies same 3-attempt tracking
- Now handles both: (1) folder doesn't exist, (2) folder is empty
- Database tracking works for both scenarios
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Remove folder creation logic in get_or_create_subfolder_path() since DAM does not allow folder creation via API. When a subfolder doesn't exist, upload to the parent folder instead of attempting to create it (which was causing 120 second timeouts).
This resolves upload failures in PROD environment during A2→A3 workflow.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit includes critical updates for PPR deployment:
1. Environment-Specific Field Mappings:
- Created field_mappings_ppr.yaml with agency code "Oliver"
- Created field_mappings_prod.yaml with agency code "0000221659"
- Updated config_loader.py to auto-detect environment based on DAM URL
- Enables seamless deployment between PPR and PROD environments
2. Metadata Extractor Enhancements:
- Added MetadataTable extraction support for nested fields
- Enables extraction of "Type of Video & Static Right" multi-value field
- Added logic to apply defaults to existing but empty fields
- Fixed agency name display_value handling for domain fields
3. Default Values Added:
- VIDEO_POST_PROD_COMPANY: "Oliver Marketing Ltd"
- AUDIO_POST_PROD_COMPANY: "Oliver Marketing Ltd"
- PROD_COMPANY (Production House): "-"
These changes ensure:
- Correct agency codes per environment (PPR/PROD)
- Proper extraction of nested tabular fields
- Default values for empty production company fields
- Seamless deployment workflow
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Key Changes:
- Updated metadata_extractor_mvp.py to use SIMPLE structure for all tabular fields
- All tabular fields now use direct value objects (no MetadataTableFieldRow wrapper)
- MAIN_LANGUAGES, ASSETCOMPLIANCE, MARKETING_TAG, CREATIVEX all use SIMPLE structure
- Master Asset ID field updated to SIMPLE structure
- Date fields now use type 'string' instead of 'long'
- Matches DAM reference structure from asset_representation.json
Added Files:
- metadata_extractor_mvp_PROD.py: PROD-specific version with same SIMPLE structure
- Backup files for safety
- Analysis and comparison documentation
Environment:
- Tested and working in PPR environment (ppr.dam.ferrero.com)
- All tabular fields match DAM-supplied reference structure
- Successful uploads confirmed
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
