Compare commits

...
Sign in to create a new pull request.

12 commits
main ... ppr

Author SHA1 Message Date
nickviljoen
de04cfc8fb Docs: Update README and CLAUDE.md with folder-only template and EOL workflow
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 21:34:06 +02:00
nickviljoen
37097d2148 Enhancement: Template-based folder-only mode and EOL workflow
Folder-only mode (-N flag) now uses asset_representation_template.json
as the base for all metadata fields, matching the exact structure the DAM
API expects. Also adds EOL (External Legal Opinion) as a new asset type
with field overrides (Agency=-, ProdCompany=-, Languages=Global,
IPRights=Yes, Licensing=No, no validity dates).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 15:52:22 +02:00
nickviljoen
2ec22c62a5 Fix: Folder-only mode metadata format for PROD DAM compatibility
Folder-only mode (-N suffix files) was sending simplified metadata that
PROD DAM rejected with "unmarshalling parameter" error. Updated to use
DomainValue format for domained fields, correct asset type field ID
(FERRERO.FIELD.MKTG.ASSET TYPE), asset type code mapping (e.g. SND→sound),
validity dates, and forced values from config.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 12:24:44 +02:00
nickviljoen
7412019053 Update: Move ELO to asset type, add VOD social media code, remove OLV asset type
- ELO (External Legal Opinion) is now a standard asset type instead of a separate
  document type flag in the filename. Field overrides (Agency, Prod Company,
  Languages) still trigger when ELO is selected as asset type.
- Added VOD to social media platform codes
- Removed OLV from asset type mappings
- Renamed document_type_overrides to asset_type_overrides throughout

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 21:21:44 +02:00
nickviljoen
7deb9db0a5 Fix: Update MAIN_LANGUAGES values array for tabular fields in DAM upload
The filename_updates logic was only updating field['value'] (singular) but for
tabular fields like MAIN_LANGUAGES, the DAM reads from field['values'] (plural
array). This caused the master's original language (e.g. "Global") to persist
instead of the correct language from the filename (e.g. "PL").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 17:27:09 +02:00
nickviljoen
ad714d4b14 Revert "Fix: Add YouTube platform mapping and social media code fallback for CreativeX"
This reverts commit e327502723.
2026-02-13 17:16:41 +02:00
nickviljoen
e327502723 Fix: Add YouTube platform mapping and social media code fallback for CreativeX
YouTube Ads was missing from the DAM-CX mappings CSV, causing empty
Platform > Rating fields for YouTube assets. Also adds a fallback that
derives the CreativeX platform from the filename social media code (e.g.
YTA -> YouTube) when the database has no mapped platforms.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 17:00:35 +02:00
nickviljoen
a2f1954038 Fix: CreativeX tracking ID fallback, filename stripping, and social media codes
CreativeX lookup now falls back to tracking ID search when filename match fails
(handles mismatched naming from CreativeX PDFs). strip_upload_components now
only removes job number and tracking ID, keeping social media codes (YTA, DV3,
etc.) in the clean filename. Updated SOCIAL_MEDIA_CODES from 4 to 39 codes
sourced from the Ferrero naming tool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 13:25:08 +02:00
nickviljoen
b89a44984d Fix: Pass notifier to process_box_file and use case-sensitive Master ID check
The notifier variable was referenced inside process_box_file but never passed
as a parameter, causing NameError for any file hitting the Master Tracking ID
check. Also changed the check from case-insensitive (.upper().startswith('M'))
to case-sensitive (.startswith('M')) to avoid false positives on random tracking
IDs like mviSv5.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 11:50:15 +02:00
nickviljoen
1b4e1a1cbc Fix: PPR MASTERASSETIDS payload updated to free text tabular field format
Changed from DomainValue structure to simple value structure per client specification.
Field is now a free text multivalue field instead of domain-based.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 15:28:37 +02:00
nickviljoen
26363f772d Enhancement: Campaign re-opening support and PPR master asset ID registration
A1→A2 now handles re-processing when campaign is reset to A1 after adding new
master assets. Existing assets reuse tracking IDs and skip Box upload, new assets
are processed normally. Also includes PPR domain registration for multiple master
asset IDs in a2_to_a3 and dam_client.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-05 21:06:14 +02:00
nickviljoen
444ac7ac6d Fix: PPR multiple master asset IDs now correctly populate MASTERASSETIDS field
Fixed issue where only 1 of 3 master asset IDs was being added to the
FERRERO.MASTERASSETIDS tabular field. The bug was caused by calling
_add_master_asset_id_field() before _add_master_asset_ids_field(),
which created the field with a single value and blocked the multi-value
method from adding all IDs.

Changes:
- metadata_extractor_mvp.py: Prioritize master_opentext_ids parameter
  using if/elif logic to prevent single-ID method from blocking multi-ID
- a2_to_a3_upload_polling.py: Load multiple master assets in PPR mode
- filename_parser.py: Parse multiple tracking IDs (e.g., ID1+ID2+ID3)
- query_db.py: Fix .env loading path
- Added documentation and test files for multiple master asset IDs

Tested in PPR with 3 tracking IDs (BqB8vo+SfUQ7m+laRJo0) - all 3 master
asset IDs now correctly appear in the metadata structure.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-03 21:02:09 +02:00
17 changed files with 2872 additions and 163 deletions

View file

@ -126,16 +126,22 @@ tail -f logs/cron_a1_a2.log
- Rejection comment extraction (A5→A6 specific)
- **metadata_extractor_mvp.py**: Field mapping and metadata transformation
- Loads 27 MVP fields from `config/field_mappings.yaml`
- Handles filename-based updates
- Loads MVP fields from environment-specific config (`field_mappings_ppr.yaml` or `field_mappings_prod.yaml`)
- **Two tracking modes:** Full inheritance (from master metadata) and folder-only (`-N` suffix, uses `config/asset_representation_template.json` as base)
- Handles filename-based updates, forced values, defaults, and asset type overrides
- Asset type overrides (e.g., EOL) can set/remove fields with final precedence
- Force-sets required values (e.g., STATE = "Local")
- Uses DomainValue format for domained fields when setting values on template fields
### Configuration Architecture
**Hierarchical config system:**
- `.env`: Environment variables (credentials, never committed)
- `config/config.yaml`: Main configuration (references .env vars)
- `config/field_mappings.yaml`: Editable field definitions (add/remove fields without code changes)
- `config/field_mappings_ppr.yaml`: PPR field definitions (auto-loaded when DAM URL contains 'ppr')
- `config/field_mappings_prod.yaml`: PROD field definitions (auto-loaded otherwise)
- `config/asset_type_mappings.yaml`: 3-letter code to DAM code mappings (e.g., EOL -> externallegalopinion)
- `config/asset_representation_template.json`: Reference template for folder-only mode (-N flag), contains full field metadata structure
- `../Box-config.json`: Box JWT credentials (one directory up from Python-Version)
**Important**: Box-config.json MUST be located at `../Box-config.json` (one folder up). This is hardcoded in config.yaml as `rsa_private_key_path: ../Box-config.json`.
@ -181,8 +187,11 @@ tail -f logs/cron_a1_a2.log
**A2→A3 (Upload from Box):**
- Polls `BOX_ROOT_FOLDER_A2_A3` (348526703108) for new files
- Parses tracking ID from filename (V2 format)
- Loads master metadata from database
- Two tracking modes:
- **Full inheritance**: Loads master metadata from database, inherits all fields
- **Folder-only** (`-N` suffix): Uses `config/asset_representation_template.json` as base, populates from filename
- Updates Description, Language, State fields from filename
- Applies asset type overrides (e.g., EOL sets Agency="-", Languages="Global", IPRights="Yes", removes validity dates)
- Deletes file from Box after successful upload
- Updates campaign status A2→A3 when ALL assets uploaded
@ -219,7 +228,10 @@ New workflow scripts should follow this pattern:
### Modifying Field Mappings
**To add/remove fields, edit `config/field_mappings.yaml`:**
**To add/remove fields, edit the environment-specific file:**
- PPR: `config/field_mappings_ppr.yaml`
- PROD: `config/field_mappings_prod.yaml`
```yaml
mvp_fields:
- FERRERO.FIELD.NEW_FIELD_NAME # Add here
@ -228,6 +240,19 @@ mvp_fields:
**No code changes required** - the system dynamically loads fields at runtime.
### Asset Type Overrides
To add field overrides for a specific asset type, add an `asset_type_overrides` section to the field mappings file:
```yaml
asset_type_overrides:
EOL: # Keyed by 3-letter asset type code
FERRERO.MARKETING.FIELD.AGENCY NAME: "-"
MAIN_LANGUAGES: "Global"
FERRERO.FIELD.ASSET VALIDITY START PERIOD: "" # Empty string removes the field
```
Overrides run after all other field processing (forced values, defaults) and take final precedence. An empty string value removes the field entirely from the payload.
### Database Queries
Common patterns used in the codebase:
@ -301,7 +326,10 @@ except Exception as e:
├── .env # Environment variables
├── config/
│ ├── config.yaml
│ ├── field_mappings.yaml
│ ├── field_mappings_ppr.yaml
│ ├── field_mappings_prod.yaml
│ ├── asset_type_mappings.yaml
│ ├── asset_representation_template.json
│ └── certificates/
│ └── dam-mtls-dev.pfx
├── database/

View file

@ -0,0 +1,378 @@
# Option 1: Multiple Tracking IDs in Filename - Implementation Guide
## Overview
Allow a single derivative/localized asset to reference multiple master assets by including multiple tracking IDs in the filename.
**Example Filename:**
```
1234568_ROC_STRANGER-THINGS_SND_6S_16x9_REF_DE_de_BqB8vo+SfUQ7m+laRJo0.jpg
^^^^^^^^^^^^^^^^^
Multiple tracking IDs
```
**Delimiter:** Use `+` to separate multiple tracking IDs (could also use `,` or `_`)
---
## Changes Required
### 1⃣ Filename Parser (`scripts/shared/filename_parser.py`)
**Current Code (line ~182):**
```python
# Tracking ID: 6 alphanumeric, optionally with -N suffix
elif re.match(r'^[a-zA-Z0-9]{6}(-N)?$', part):
tracking = part
tracking_mode = 'full'
base_tracking_id = tracking
if tracking.endswith('-N'):
tracking_mode = 'folder_only'
base_tracking_id = tracking[:-2] # Strip -N suffix
parsed['tracking_id'] = base_tracking_id
parsed['tracking_mode'] = tracking_mode
parsed['tracking_id_with_suffix'] = tracking
logger.debug("Found tracking ID: {}".format(tracking))
index += 1
```
**Modified Code:**
```python
# Tracking ID(s): 6 alphanumeric, optionally with -N suffix
# Supports multiple IDs separated by + (e.g., "BqB8vo+SfUQ7m+laRJo0")
elif re.match(r'^[a-zA-Z0-9]{6}(-N)?(\+[a-zA-Z0-9]{6}(-N)?)*$', part):
tracking_ids = []
tracking_modes = []
tracking_ids_with_suffix = []
# Split by + delimiter to get all tracking IDs
id_parts = part.split('+')
for tracking in id_parts:
tracking_mode = 'full'
base_tracking_id = tracking
if tracking.endswith('-N'):
tracking_mode = 'folder_only'
base_tracking_id = tracking[:-2] # Strip -N suffix
logger.info("Detected folder-only tracking ID: {} (base: {})".format(tracking, base_tracking_id))
tracking_ids.append(base_tracking_id)
tracking_modes.append(tracking_mode)
tracking_ids_with_suffix.append(tracking)
# Store primary (first) tracking ID for backward compatibility
parsed['tracking_id'] = tracking_ids[0]
parsed['tracking_mode'] = tracking_modes[0]
parsed['tracking_id_with_suffix'] = tracking_ids_with_suffix[0]
# Store all tracking IDs for multi-master support
parsed['tracking_ids'] = tracking_ids
parsed['tracking_modes'] = tracking_modes
parsed['tracking_ids_with_suffix'] = tracking_ids_with_suffix
parsed['has_multiple_masters'] = len(tracking_ids) > 1
logger.debug("Found {} tracking ID(s): {}".format(len(tracking_ids), ', '.join(tracking_ids)))
index += 1
```
**Key Changes:**
- Updated regex to match multiple IDs: `^[a-zA-Z0-9]{6}(-N)?(\+[a-zA-Z0-9]{6}(-N)?)*$`
- Split on `+` delimiter
- Store primary ID for backward compatibility
- Add new fields: `tracking_ids`, `has_multiple_masters`
---
### 2⃣ A2→A3 Upload Script (`scripts/a2_to_a3_upload_polling.py`)
**Current Code (line ~97):**
```python
# 2. Load master metadata from database
master_asset = db.get_master_asset(tracking_id)
if not master_asset:
raise ValueError("No master asset for tracking ID: {}".format(tracking_id))
```
**Modified Code:**
```python
# 2. Load master metadata from database (support multiple tracking IDs)
tracking_ids = parsed.get('tracking_ids', [tracking_id]) # Get all tracking IDs or fallback to single
has_multiple_masters = parsed.get('has_multiple_masters', False)
# Load all master assets
master_assets = []
master_opentext_ids = []
if has_multiple_masters:
logger.info("Multiple master assets detected: {}".format(', '.join(tracking_ids)))
for tid in tracking_ids:
master = db.get_master_asset(tid)
if not master:
logger.warning("Master asset not found for tracking ID: {}".format(tid))
continue
master_assets.append(master)
master_opentext_ids.append(master['opentext_id'])
if not master_assets:
raise ValueError("No master assets found for tracking IDs: {}".format(', '.join(tracking_ids)))
# Use first master for metadata inheritance (could enhance this later)
master_asset = master_assets[0]
logger.info("Using primary master {} for metadata, linking all {} masters".format(
tracking_ids[0], len(master_assets)))
else:
# Single master (backward compatible)
master_asset = db.get_master_asset(tracking_id)
if not master_asset:
raise ValueError("No master asset for tracking ID: {}".format(tracking_id))
master_opentext_ids = [master_asset['opentext_id']]
```
**Current Code (line ~194):**
```python
asset_rep = mvp_extractor.build_mvp_asset_representation(
master_metadata=master_asset['full_metadata'],
clean_filename=clean_filename,
parsed_filename=parsed,
box_metadata=box_metadata,
tracking_mode=tracking_mode,
master_opentext_id=master_asset['opentext_id'] # Single ID
)
```
**Modified Code:**
```python
# Pass all master opentext IDs (support multiple)
asset_rep = mvp_extractor.build_mvp_asset_representation(
master_metadata=master_asset['full_metadata'],
clean_filename=clean_filename,
parsed_filename=parsed,
box_metadata=box_metadata,
tracking_mode=tracking_mode,
master_opentext_id=master_asset['opentext_id'], # Primary for ARTESIA.FIELD.ASSET_ID
master_opentext_ids=master_opentext_ids # All IDs for MASTERASSETIDS field
)
```
**Key Changes:**
- Extract multiple tracking IDs from parsed data
- Look up all master assets in database
- Collect all master opentext_ids
- Pass list to metadata extractor
---
### 3⃣ Metadata Extractor (`scripts/shared/metadata_extractor_mvp.py`)
**Current Method Signature (line ~97):**
```python
def build_mvp_asset_representation(self, master_metadata, clean_filename,
parsed_filename, box_metadata=None,
tracking_mode='full', master_opentext_id=None):
```
**Modified Method Signature:**
```python
def build_mvp_asset_representation(self, master_metadata, clean_filename,
parsed_filename, box_metadata=None,
tracking_mode='full', master_opentext_id=None,
master_opentext_ids=None):
```
**Current Code (line ~139):**
```python
if master_opentext_id:
mvp_fields = self._add_master_asset_id_field(mvp_fields, master_opentext_id)
logger.info("Added Master Asset ID field: {}".format(master_opentext_id))
```
**Modified Code:**
```python
# Add Master Asset ID field(s) if provided (derivative tracking)
if master_opentext_id:
mvp_fields = self._add_master_asset_id_field(mvp_fields, master_opentext_id)
logger.info("Added Master Asset ID field: {}".format(master_opentext_id))
# Add MASTERASSETIDS tabular field with all master IDs (support multiple)
if master_opentext_ids and len(master_opentext_ids) > 0:
mvp_fields = self._add_master_asset_ids_field(mvp_fields, master_opentext_ids)
logger.info("Added MASTERASSETIDS field with {} value(s)".format(len(master_opentext_ids)))
```
**New Method (add after `_add_master_asset_id_field`):**
```python
def _add_master_asset_ids_field(self, mvp_fields, master_opentext_ids):
"""
Add FERRERO.MASTERASSETIDS tabular field with multiple master asset IDs
Supports Many-to-Many relationship between derivatives and masters
Args:
mvp_fields: List of MVP fields
master_opentext_ids: List of DAM Asset IDs of master assets
Returns:
Updated mvp_fields list with FERRERO.MASTERASSETIDS
"""
if not master_opentext_ids or len(master_opentext_ids) == 0:
logger.info("No master_opentext_ids provided - skipping FERRERO.MASTERASSETIDS field")
return mvp_fields
# Check if field already exists
for field in mvp_fields:
if self._get_field_id(field) == 'FERRERO.MASTERASSETIDS':
logger.info("FERRERO.MASTERASSETIDS already present")
return mvp_fields
# Build values array with all master asset IDs
values = []
for master_id in master_opentext_ids:
values.append({
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'field_value': {
'type': 'string',
'value': master_id
},
'type': 'com.artesia.metadata.DomainValue'
}
})
# Create tabular field
new_field = {
'id': 'FERRERO.MASTERASSETIDS',
'parent_table_id': 'FERRERO.TABULAR.FIELD.MASTERASSETIDS',
'type': 'com.artesia.metadata.MetadataTableField',
'values': values
}
mvp_fields.append(new_field)
logger.info("Added FERRERO.MASTERASSETIDS field with {} master asset ID(s): {}".format(
len(values), ', '.join(master_opentext_ids[:3]) + ('...' if len(master_opentext_ids) > 3 else '')))
return mvp_fields
```
**Key Changes:**
- Add `master_opentext_ids` parameter (list)
- New method `_add_master_asset_ids_field` that accepts a list
- Builds `values` array with all master IDs
- Backward compatible (still uses single `master_opentext_id` for ARTESIA.FIELD.ASSET_ID)
---
## Testing Examples
### Single Master (Backward Compatible)
**Filename:** `1234568_ROC_STRANGER-THINGS_SND_6S_16x9_REF_DE_de_BqB8vo.jpg`
**Parsed:**
```python
{
'tracking_id': 'BqB8vo',
'tracking_ids': ['BqB8vo'],
'has_multiple_masters': False
}
```
**Result:** Single ID in MASTERASSETIDS field (current behavior)
---
### Multiple Masters (New Feature)
**Filename:** `1234568_ROC_STRANGER-THINGS_SND_6S_16x9_REF_DE_de_BqB8vo+SfUQ7m+laRJo0.jpg`
**Parsed:**
```python
{
'tracking_id': 'BqB8vo', # Primary (for backward compatibility)
'tracking_ids': ['BqB8vo', 'SfUQ7m', 'laRJo0'],
'has_multiple_masters': True
}
```
**Database Lookups:**
- BqB8vo → fc5c389776516bb58044c7d4bf479da458599baf
- SfUQ7m → ad3948d72ea8550a338a600ae87a1bdd1968b066
- laRJo0 → 020d76f957ec9f4ec0b18035a2d012cd3fd376c2
**Result:** 3 IDs in MASTERASSETIDS field values array
---
## Migration Path
1. **Phase 1 - Implement Code** (No Breaking Changes)
- Add changes to all 3 files
- Test with single tracking ID (should work exactly as before)
- Backward compatible with existing filenames
2. **Phase 2 - Test Multiple IDs**
- Create test file with multiple tracking IDs
- Upload to PPR with `--dryrun`
- Verify 3 values in MASTERASSETIDS field
3. **Phase 3 - Agency Tool Integration**
- Agency tool generates filenames with `+` delimiter
- Agency tool uses multiple tracking IDs when needed
- Most files will still have single tracking ID (normal case)
4. **Phase 4 - Production Deployment**
- Enable in PROD after testing in PPR
- Update field in PROD DAM schema first
- Deploy code changes
---
## Alternative Delimiters
If `+` causes issues, alternatives:
| Delimiter | Example | Notes |
|-----------|---------|-------|
| `+` | `BqB8vo+SfUQ7m` | ✅ Recommended (clear separator) |
| `,` | `BqB8vo,SfUQ7m` | ⚠️ Might conflict with CSV exports |
| `_` | `BqB8vo_SfUQ7m` | ⚠️ Already used in filename structure |
| `~` | `BqB8vo~SfUQ7m` | ✅ Alternative if + causes issues |
---
## Error Handling
**What happens if one tracking ID is not found?**
```python
# Option A: Skip missing masters (log warning)
for tid in tracking_ids:
master = db.get_master_asset(tid)
if not master:
logger.warning("Master asset not found for tracking ID: {}".format(tid))
continue # Skip this one, continue with others
# Option B: Fail entire upload (strict)
for tid in tracking_ids:
master = db.get_master_asset(tid)
if not master:
raise ValueError("Master asset not found for tracking ID: {}".format(tid))
```
**Recommendation:** Use Option A (skip missing) - derivative still uploads with available master links.
---
## Summary
**Files to Modify:**
1. `scripts/shared/filename_parser.py` - Parse multiple tracking IDs
2. `scripts/a2_to_a3_upload_polling.py` - Look up multiple masters
3. `scripts/shared/metadata_extractor_mvp.py` - Add all IDs to field
**Backward Compatible:** ✅ Yes - existing single-ID filenames work exactly as before
**Ready to Implement:** This document provides all code changes needed.

View file

@ -0,0 +1,179 @@
# PPR-Only Multiple Tracking IDs - Implementation Complete
## ✅ Changes Implemented
Multiple tracking IDs feature is now **ACTIVE in PPR** and **DISABLED in PROD** via environment detection.
---
## Files Modified
### 1. `scripts/shared/filename_parser.py`
- Added `__init__` method with DAM URL parameter
- Added `_is_ppr_environment()` method
- Updated tracking ID parsing to:
- **PPR**: Parse multiple IDs separated by `+` (e.g., `BqB8vo+SfUQ7m+laRJo0`)
- **PROD**: Use only first ID (backward compatible)
### 2. `scripts/a2_to_a3_upload_polling.py`
- Pass DAM URL to FilenameParser for environment detection
- Loop through all tracking IDs (PPR) or single ID (PROD)
- Look up all master assets in database
- Collect all `opentext_id` values
- Pass list to metadata extractor
### 3. `scripts/shared/metadata_extractor_mvp.py`
- Added `master_opentext_ids` parameter (list)
- New method: `_add_master_asset_ids_field()` to handle multiple IDs
- Builds `values` array with all master IDs
---
## Environment Detection Logic
**PPR Environment:**
- DAM URL contains: `ppr.dam.ferrero.com`
- Multiple tracking IDs: ✅ **ENABLED**
- Filename format: `1234568_ROC_ST_SND_6S_16x9_REF_DE_de_BqB8vo+SfUQ7m.jpg`
**PROD Environment:**
- DAM URL contains: `dam.ferrero.com` (not ppr)
- Multiple tracking IDs: ❌ **DISABLED**
- Filename format: `1234568_ROC_ST_SND_6S_16x9_REF_DE_de_BqB8vo.jpg` (single ID)
- If multiple IDs provided, uses FIRST ID only with warning
---
## Testing in PPR
### Test 1: Single Tracking ID (Backward Compatible)
**Filename:**
```
1234568_ROC_STRANGER-THINGS_SND_6S_16x9_REF_DE_de_BqB8vo.jpg
```
**Expected Result:**
- Parses as single tracking ID
- One master asset looked up
- One value in MASTERASSETIDS field
- ✅ Works exactly as before
### Test 2: Multiple Tracking IDs (New Feature)
**Filename:**
```
1234568_ROC_STRANGER-THINGS_SND_6S_16x9_REF_DE_de_BqB8vo+SfUQ7m+laRJo0.jpg
```
**Expected Result:**
- PPR environment detected
- Parses 3 tracking IDs: `BqB8vo`, `SfUQ7m`, `laRJo0`
- Looks up 3 master assets in database
- Gets 3 opentext_ids:
- `fc5c389776516bb58044c7d4bf479da458599baf`
- `ad3948d72ea8550a338a600ae87a1bdd1968b066`
- `020d76f957ec9f4ec0b18035a2d012cd3fd376c2`
- Creates MASTERASSETIDS field with 3 values
**Log Output:**
```
PPR Environment - Multiple tracking IDs detected: 3
Parsed 3 tracking IDs: BqB8vo, SfUQ7m, laRJo0
PPR - Multiple master assets detected: BqB8vo, SfUQ7m, laRJo0
Using primary master BqB8vo for metadata, linking 3 total masters
PPR - Added MASTERASSETIDS field with 3 master IDs
Added FERRERO.MASTERASSETIDS field with 3 master asset ID(s): fc5c389776516bb58044c7d4bf479da458599baf, ad3948d72ea8550a338a600ae87a1bdd1968b066, 020d76f957ec9f4ec0b18035a2d012cd3fd376c2
```
---
## Test Commands
### Dry Run (Recommended First)
```bash
python scripts/a2_to_a3_upload_polling.py --dryrun
```
Check the JSON output for:
```json
{
"id": "FERRERO.MASTERASSETIDS",
"parent_table_id": "FERRERO.TABULAR.FIELD.MASTERASSETIDS",
"type": "com.artesia.metadata.MetadataTableField",
"values": [
{"value": {"field_value": {"value": "fc5c389776516bb58044c7d4bf479da458599baf"}}},
{"value": {"field_value": {"value": "ad3948d72ea8550a338a600ae87a1bdd1968b066"}}},
{"value": {"field_value": {"value": "020d76f957ec9f4ec0b18035a2d012cd3fd376c2"}}}
]
}
```
### Real Upload to PPR
```bash
python scripts/a2_to_a3_upload_polling.py
```
Then verify in PPR DAM:
1. Search for the uploaded asset
2. Open metadata
3. Check "Master Asset IDs" tabular field
4. Should show multiple rows
---
## Error Handling
**Missing Master Assets:**
- If one tracking ID is not found in database, it's skipped with warning
- Derivative still uploads with available master links
- Log message: `Master asset not found for tracking ID: xyz - skipping`
**PROD Environment with Multiple IDs:**
- Uses FIRST tracking ID only
- Logs warning: `PROD Environment - Multiple tracking IDs not supported, using first ID only`
- Works as backward compatible (no errors)
---
## Current Environment Check
Your `.env` file shows:
```
DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
```
**PPR Environment** - Multiple tracking IDs are **ENABLED**
---
## Agency Tool Requirements
To use multiple tracking IDs, the Agency tool needs to:
1. Concatenate tracking IDs with `+` delimiter
2. Example: `tracking_id_1 + "+" + tracking_id_2 + "+" + tracking_id_3`
3. Place in filename: `{job}_{brand}_{...}_{tracking_ids}.{ext}`
**Most derivatives will still use single tracking ID** - this is only for special cases where one derivative references multiple masters.
---
## Production Safety
✅ **PROD is Protected:**
- Environment detection prevents multiple IDs in PROD
- If multiple IDs accidentally used, only first ID is processed
- No breaking changes to PROD behavior
- Fully backward compatible
---
## Ready to Test! 🚀
Your PPR environment is now ready to test multiple tracking IDs.
1. Create test file with multiple IDs
2. Upload to Box: `DAM-UPLOAD/1234568/`
3. Run with `--dryrun` first
4. Verify JSON shows multiple values
5. Real upload and check in PPR DAM

View file

@ -2,8 +2,8 @@
**Complete automated workflow for Ferrero DAM Content Scaling**
**Version:** 2.0
**Last Updated:** November 5, 2025
**Version:** 2.1
**Last Updated:** March 31, 2026
**Status:** ✅ Production Ready & Fully Tested
---
@ -334,6 +334,12 @@ crontab -e
7. Delete file from Box
8. **Update A2→A3 when ALL campaign assets uploaded**
**Two tracking modes:**
- **Full inheritance** (standard): Inherits all metadata from the master asset
- **Folder-only** (`-N` suffix): Builds metadata from a reference template (`config/asset_representation_template.json`) and populates values from the filename. Used when the derivative only needs the upload folder from the master.
**Asset type overrides:** Certain asset types (e.g., EOL) trigger field overrides configured in the environment's field mappings file (e.g., Agency Name, Languages, IP Rights, validity dates).
**Box Folder:** 348526703108 (Agency Uploads)
**Email:** a2_to_a3_file_uploaded, a2_to_a3_complete
@ -946,6 +952,35 @@ scp Box-config.json user@server:/opt/ferrero-automation/
scp -r Python-Version/ user@server:/opt/ferrero-automation/
```
### Field Mappings (Environment-Specific)
The system auto-detects the environment from the DAM URL and loads the appropriate config:
- **PPR:** `config/field_mappings_ppr.yaml` (pre-production, `ppr.dam.ferrero.com`)
- **PROD:** `config/field_mappings_prod.yaml` (production, `dam.ferrero.com`)
Each file defines: MVP fields, filename update rules, forced values, defaults, and asset type overrides.
### Asset Type Mappings
`config/asset_type_mappings.yaml` maps 3-letter codes from the naming tool to DAM domain values (e.g., `EHI` -> `heroimage`, `EOL` -> `externallegalopinion`).
### Asset Representation Template
`config/asset_representation_template.json` is the reference template for folder-only mode (`-N` flag uploads). It contains the full field metadata structure that the DAM API requires for asset creation. This template was provided by the client and should be updated if the DAM metadata model changes.
### Asset Type Overrides (EOL Example)
Certain asset types trigger field overrides configured in the field mappings file. For example, **EOL (External Legal Opinion)** overrides:
- Agency Name = "-"
- Production House = "-"
- Main Languages = "Global"
- IP Rights = "Yes"
- Licensing = "No"
- Validity dates removed
These overrides are applied after all other field processing and take final precedence.
---
## 🗄️ Database

View file

@ -0,0 +1,830 @@
{
"asset_resource": {
"asset": {
"metadata": {
"metadata_element_list": [
{
"column_name": "ASSET_TYPE",
"data_length": 30,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.MARKETING.ASSETTYPE",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.MKTG.ASSET TYPE",
"multilingual": false,
"name": "Asset Type",
"prompt": "Asset Type",
"required": false,
"restriction_id": 7242,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "FISCAL__YEAR",
"data_length": 100,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO_DOMAIN_FISCAL_YEAR",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.FISCAL YEAR",
"multilingual": false,
"name": "Release Fiscal Year",
"prompt": "Release Fiscal Year",
"required": true,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "ECOMMERCE",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "DESCR",
"data_length": 249,
"data_type": "CHAR",
"description": "Descriptive information",
"displayable": true,
"domained": false,
"edit_type": "TEXTAREA",
"editable": true,
"enabled": true,
"facetable": false,
"id": "ARTESIA.FIELD.ASSET DESCRIPTION",
"multilingual": false,
"name": "Description",
"prompt": "Description",
"required": false,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "UOIS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"value": {
"type": "string",
"value": ""
}
}
},
{
"column_name": "FLAVOUR",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.MARKETING.FLAVOUR",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.MARKETING.FLAVOUR",
"multilingual": false,
"name": "Flavour",
"prompt": "Flavour",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "SIZE",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.MARKETING.SIZE",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.MARKETING.SIZE",
"multilingual": false,
"name": "Size",
"prompt": "Size",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "ASSET_STATE",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.GLOBAL.LOCAL",
"domained": true,
"edit_type": "COMBO_NOTNULL",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.STATE",
"multilingual": false,
"name": "Global/Local",
"prompt": "Global/Local",
"required": true,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "FERRERO_IC_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "NAME",
"data_length": 256,
"data_type": "CHAR",
"description": "Original name of master object",
"displayable": true,
"domained": false,
"edit_type": "SIMPLE",
"editable": false,
"enabled": true,
"facetable": false,
"id": "ARTESIA.FIELD.ASSET NAME",
"multilingual": false,
"name": "Asset Name",
"prompt": "Asset Name",
"required": false,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "UOIS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"value": {
"type": "string",
"value": ""
}
}
},
{
"column_name": "SUB_BRAND",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.SUBBRAND",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.SUB BRAND",
"multilingual": false,
"name": "Sub-Brands",
"prompt": "Sub-Brands",
"required": false,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "ECOMMERCE",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "VALIDATION_STARTING_DATE",
"data_length": 20,
"data_type": "DATE",
"displayable": true,
"domained": false,
"edit_type": "DATE",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.ASSET VALIDITY START PERIOD",
"multilingual": false,
"name": "Asset validity start period",
"prompt": "Asset validity start period",
"required": false,
"restriction_id": 7242,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "FERRERO_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"value": {
"type": "string",
"value": ""
}
}
},
{
"column_name": "VALIDATION_ENDING_DATE",
"data_length": 20,
"data_type": "DATE",
"displayable": true,
"domained": false,
"edit_type": "DATE",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.FIELD.ASSET VALIDITY END PERIOD",
"multilingual": false,
"name": "Asset validity end period",
"prompt": "Asset validity end period",
"required": false,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "FERRERO_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"value": {
"type": "string",
"value": ""
}
}
},
{
"column_name": "AGENCY_NAME",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKETING.AGENCY_NAME",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.AGENCY NAME",
"multilingual": false,
"name": "Agency Name",
"prompt": "Agency Name",
"required": false,
"restriction_id": 7336,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "CREATIVE_LINK",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "TEXTAREA",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.FIELD.CREATIVEX LINK",
"multilingual": false,
"name": "CreativeX Hyperlink",
"prompt": "CreativeX Hyperlink",
"required": false,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "FERRERO_ASSET_CREATIVEX",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"value": {
"type": "string",
"value": ""
}
}
},
{
"column_name": "IP_RIGHTS",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKETING.IPRIGHTS",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": true,
"id": "FERRERO.MARKET.FIELD.IPRIGHT",
"multilingual": false,
"name": "IP Rights",
"prompt": "IP Rights",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "TOTAL_BUYOUT",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKETING.TOTAL_BUYOUT",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.FIELD.BUYOUT",
"multilingual": false,
"name": "Touchpoint Scope",
"prompt": "Touchpoint Scope",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "FERRERO_PROPERTY",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKETING.FERRERO_PROPERTY",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.FIELD.FERRERO PROPERTY",
"multilingual": false,
"name": "Ferrero Property",
"prompt": "Ferrero Property",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "VID_AND_STAT_RIGHT",
"data_length": 100,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKET.TECH_VALID",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.VID_N_STAT",
"multilingual": false,
"name": "Video and Static Right",
"prompt": "Video and Static Right",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "PRODUCTION_COMPANY",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.DOMAIN.MARKETING.PRODUCT",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.PROD_COMPANY",
"multilingual": false,
"name": "Production House",
"prompt": "Production House",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "LICENSING",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKET.TECH_VALID",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.FIELD.LICENSIN",
"multilingual": false,
"name": "Licensing",
"prompt": "Licensing",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"cascading_group_id": "FERRERO.MARKET.CG.LICENSE",
"column_name": "LICENSOR",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "CASCADING",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKET.FIELD.LICENSE",
"multilingual": false,
"name": "License",
"prompt": "License",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "SPOT_VERSION",
"data_length": 100,
"data_type": "CHAR",
"displayable": true,
"domain_id": "FERRERO.MARKETING.SPOT_VERSION",
"domained": true,
"edit_type": "COMBO",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.SPOT_VERSION",
"multilingual": false,
"name": "Spot Version",
"prompt": "Spot Version",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "DIRECTOR_NAME",
"data_length": 100,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "SIMPLE",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.DIRECTOR_NAME",
"multilingual": false,
"name": "Director Name",
"prompt": "Director Name",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "VIDEO_POST_PRODUCTION_COMPANY",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "SIMPLE",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.VIDEO_POST_PROD_COMPANY",
"multilingual": false,
"name": "Video Post-Production Company",
"prompt": "Video Post-Production Company",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "VIDEO_COMPANY_DETAILS",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "TEXTAREA",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.VID_POST_PROD_CONTACT",
"multilingual": false,
"name": "Video Post Production Company Contact Details",
"prompt": "Video Post Production Company Contact Details",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "AUDIO_POST_PRODUCTION_COMPANY",
"data_length": 200,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "SIMPLE",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.AUDIO_POST_PROD_COMPANY",
"multilingual": false,
"name": "Audio Post-Production Company",
"prompt": "Audio Post-Production Company",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"column_name": "AUDIO_COMPANY_DETAILS",
"data_length": 2000,
"data_type": "CHAR",
"displayable": true,
"domained": false,
"edit_type": "TEXTAREA",
"editable": true,
"enabled": true,
"facetable": false,
"id": "FERRERO.MARKETING.FIELD.AUDIO_POST_PROD_CONTACT",
"multilingual": false,
"name": "Audio Post Production Company Contact Details",
"prompt": "Audio Post Production Company Contact Details",
"required": false,
"restriction_id": 5429,
"scale": 0,
"searchable": true,
"searchable_scope_id": "1",
"searchable_scope_num_id": 1,
"sortable": true,
"system_field": false,
"table_name": "MARKETING_ASSET_DETAILS",
"trigger_field": false,
"type": "com.artesia.metadata.MetadataField",
"value": {
"cascading_domain_value": false,
"domain_value": false,
"is_locked": false
}
},
{
"id": "MAIN_LANGUAGES",
"parent_table_id": "FERRERO.TABULAR.FIELD.MAIN LANGUAGES",
"type": "com.artesia.metadata.MetadataTableField",
"values": []
},
{
"id": "FERRERO.FIELD.ASSETCOMPLIANCE",
"parent_table_id": "FERRERO.TABULAR.FIELD.ASSETCOMPLIANCE",
"type": "com.artesia.metadata.MetadataTableField",
"values": []
},
{
"id": "MARKETING_TAG",
"parent_table_id": "FERRERO.TABULAR.FIELD.MARKETING_TAG",
"type": "com.artesia.metadata.MetadataTableField",
"values": []
},
{
"id": "FERRERO.MARKET.FIELD.TYPE_VID",
"parent_table_id": "FERRERO.TABULAR.VID_STAT_TYPE",
"type": "com.artesia.metadata.MetadataTableField",
"values": []
}
]
},
"metadata_model_id": "ECOMMERCE",
"security_policy_list": [
{
"id": 1594
}
]
}
}
}

View file

@ -33,7 +33,6 @@ FPO: keyvisual # Front of Pack Image (alias for IMG)
LGL: localguidelines # Local Guidelines
LOG: ferrerologo # Logo
MLF: marketingleaflet # Marketing Leaflet
OLV: onlinevideodigitalvideo # On Line Video
PAW: packartworks # Pack Artworks
PKI: packshot # Pack Images (was packshot)
POS: posm # POS Material
@ -46,5 +45,8 @@ SGL: licenseshighlights # Styleguide Licenses
TVC: tvc # TVC
VIE: visualidentityelements # Visual Identity Elements
# External Legal Opinion
EOL: externallegalopinion # External Legal Opinion (triggers field overrides)
# Note: If a 3-letter code is not in this mapping, it will be passed through as-is
# and may fail DAM validation if the code doesn't exist in DAM's domain

View file

@ -76,3 +76,16 @@ defaults:
FERRERO.MARKETING.FIELD.VIDEO_POST_PROD_COMPANY: "Oliver Marketing Ltd"
FERRERO.MARKETING.FIELD.AUDIO_POST_PROD_COMPANY: "Oliver Marketing Ltd"
FERRERO.MARKET.PROD_COMPANY: "-" # Production House
# Asset type overrides (keyed by 3-letter asset type code)
# Applied AFTER normal field updates and forced values
# Overrides specific fields when a matching asset type is detected in the filename
asset_type_overrides:
EOL: # External Legal Opinion - selected as asset type in naming tool
FERRERO.MARKETING.FIELD.AGENCY NAME: "-"
FERRERO.MARKET.PROD_COMPANY: "-"
MAIN_LANGUAGES: "Global"
FERRERO.MARKET.FIELD.IPRIGHT: "Yes"
FERRERO.MARKET.FIELD.LICENSIN: "No"
FERRERO.FIELD.ASSET VALIDITY START PERIOD: "" # Remove validity dates for EOL
FERRERO.FIELD.ASSET VALIDITY END PERIOD: "" # Remove validity dates for EOL

View file

@ -4,8 +4,9 @@ import sys
import psycopg2
from dotenv import load_dotenv
# Load env vars
load_dotenv('/Users/daveporter/Desktop/CODING-2024/Ferrero-Opentext/Python-Version/.env')
# Load env vars from current directory
script_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(os.path.join(script_dir, '.env'))
try:
conn = psycopg2.connect(

View file

@ -216,16 +216,68 @@ def process_campaign(campaign, dam, box, db, notifier, config):
else:
logger.info("Processing: {}".format(asset_name))
# 1. Download from DAM
# 1. Extract Global Campaign Reference (needed for tracking ID lookup)
global_ref = db.extract_global_campaign_reference(asset, campaign_number)
# 2. Find existing tracking ID or generate new one
# Handles re-processing: if campaign was reset to A1 after adding new masters,
# existing assets keep their tracking IDs, new assets get new IDs
tracking_result = db.find_or_create_tracking_id(
opentext_id=asset_id,
local_campaign_id=global_ref['local_campaign_id']
)
tracking_id = tracking_result['tracking_id']
is_existing = tracking_result['is_existing']
if is_existing:
# Asset already processed in a previous A1→A2 cycle
existing_master = db.get_master_asset(tracking_id)
if existing_master and existing_master.get('box_file_id'):
logger.info("Re-processing: reusing tracking ID {} for existing asset {} (skipping download/upload)".format(
tracking_id, asset_name))
box_result = {
'file_id': existing_master['box_file_id'],
'url': existing_master['box_url']
}
# Update database metadata (asset_data may have changed in DAM)
db_result = db.store_master_asset(
tracking_id=tracking_id,
opentext_id=asset_id,
asset_data=asset,
box_file_id=box_result['file_id'],
box_url=box_result['url'],
upload_folder_id=final_folder_id,
global_master_campaign_id=global_ref['global_master_campaign_id'],
global_master_folder_id=global_ref['global_master_folder_id'],
local_campaign_id=global_ref['local_campaign_id']
)
if db_result['success']:
processed_assets.append({
'asset_id': asset_id,
'asset_name': asset_name,
'tracking_id': tracking_id,
'box_file_id': box_result['file_id'],
'box_url': box_result['url'],
'is_existing': True
})
logger.info("✓ Existing asset confirmed: {}{} (skipped)".format(asset_name, tracking_id))
else:
raise Exception("Database update failed for existing asset")
continue # Skip to next asset
else:
# Tracking ID exists but no usable Box info - process normally
logger.info("Existing tracking ID {} found but no Box info - downloading/uploading".format(tracking_id))
# 3. Download from DAM (new assets or existing without Box info)
file_path = dam.download_asset(
asset_id,
output_dir='temp/downloads/{}'.format(campaign_id)
)
# 2. Generate tracking ID (regular files never start with 'M')
tracking_id = db.generate_unique_tracking_id(is_master=False)
# 3. Upload to Box (preserve folder structure from DAM)
# 4. Upload to Box (preserve folder structure from DAM)
box_result = box.upload_with_tracking_id(
file_path=file_path,
campaign_id=campaign_number,
@ -234,9 +286,6 @@ def process_campaign(campaign, dam, box, db, notifier, config):
subfolder_path=folder_path
)
# 4. Extract Global Campaign Reference and Local Campaign ID
global_ref = db.extract_global_campaign_reference(asset, campaign_number)
# 5. Store in database
db_result = db.store_master_asset(
tracking_id=tracking_id,
@ -275,9 +324,10 @@ def process_campaign(campaign, dam, box, db, notifier, config):
'asset_name': asset_name,
'tracking_id': tracking_id,
'box_file_id': box_result['file_id'],
'box_url': box_result['url']
'box_url': box_result['url'],
'is_existing': False
})
logger.info("Success: {}{}".format(asset_name, tracking_id))
logger.info("New asset: {}{}".format(asset_name, tracking_id))
else:
raise Exception("Database storage failed")
@ -295,10 +345,17 @@ def process_campaign(campaign, dam, box, db, notifier, config):
# CHECK: All assets processed successfully?
all_done = len(processed_assets) == total_assets
# Count new vs existing assets
new_assets = [a for a in processed_assets if not a.get('is_existing')]
existing_assets = [a for a in processed_assets if a.get('is_existing')]
logger.info("")
logger.info("Campaign {} Results:".format(campaign_id))
logger.info(" Total: {}".format(total_assets))
logger.info(" Successful: {}".format(len(processed_assets)))
if existing_assets:
logger.info(" - New assets: {}".format(len(new_assets)))
logger.info(" - Existing assets (skipped download/upload): {}".format(len(existing_assets)))
logger.info(" Failed: {}".format(len(failed_assets)))
logger.info(" All Done: {}".format("YES" if all_done else "NO"))
logger.info("")
@ -341,19 +398,18 @@ def process_campaign(campaign, dam, box, db, notifier, config):
os.makedirs("temp")
with open(csv_path, 'w', newline='') as csvfile:
fieldnames = ['Filename', 'Tracking ID', 'Campaign Number']
fieldnames = ['Filename', 'Tracking ID', 'Campaign Number', 'Status']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
for asset in processed_assets:
# 2024-03-22: Clean filename request (remove tracking ID)
# Assuming tracking ID is at the end or we just want the asset_name
clean_name = asset['asset_name'] # asset_name from db.store_master_asset is typically used
clean_name = asset['asset_name']
writer.writerow({
'Filename': clean_name,
'Tracking ID': asset['tracking_id'],
'Campaign Number': campaign_number
'Campaign Number': campaign_number,
'Status': 'Existing' if asset.get('is_existing') else 'New'
})
logger.info("Generated CSV report: {}".format(csv_path))
@ -372,6 +428,8 @@ def process_campaign(campaign, dam, box, db, notifier, config):
'campaign_id': campaign_id,
'campaign_number': campaign_number,
'asset_count': len(processed_assets),
'new_asset_count': len(new_assets),
'existing_asset_count': len(existing_assets),
'processed_assets': processed_assets
},
attachments=attachments

View file

@ -57,7 +57,7 @@ logging.basicConfig(
logger = logging.getLogger('A2toA3')
def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, keep_files=False, dryrun=False):
def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, notifier, keep_files=False, dryrun=False):
"""
Process a single file from Box folder
@ -93,11 +93,42 @@ def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, kee
if subfolder_path:
logger.info("From Box subfolder: {} -> will create in DAM".format(subfolder_path))
# 2. Load master metadata from database
master_asset = db.get_master_asset(tracking_id)
# 2. Load master metadata from database (support multiple tracking IDs in PPR)
tracking_ids = parsed.get('tracking_ids', [tracking_id]) # Get all IDs or fallback to single
has_multiple_masters = parsed.get('has_multiple_masters', False)
# CHECK: Warn if Master Tracking ID is used (starts with M)
if tracking_id.upper().startswith('M'):
# Load all master assets (PPR: multiple, PROD: single)
master_assets = []
master_opentext_ids = []
if has_multiple_masters:
logger.info("PPR - Multiple master assets detected: {}".format(', '.join(tracking_ids)))
for tid in tracking_ids:
master = db.get_master_asset(tid)
if not master:
logger.warning("Master asset not found for tracking ID: {} - skipping".format(tid))
continue
master_assets.append(master)
master_opentext_ids.append(master['opentext_id'])
if not master_assets:
raise ValueError("No master assets found for tracking IDs: {}".format(', '.join(tracking_ids)))
# Use first master for metadata inheritance
master_asset = master_assets[0]
logger.info("Using primary master {} for metadata, linking {} total masters".format(
tracking_ids[0], len(master_assets)))
else:
# Single master (backward compatible)
master_asset = db.get_master_asset(tracking_id)
if not master_asset:
# Will check below
master_asset = None
else:
master_opentext_ids = [master_asset['opentext_id']]
# CHECK: Warn if Master Tracking ID is used (starts with uppercase M)
if tracking_id.startswith('M'):
logger.warning("Detected Master Tracking ID in Version/Derivative upload folder: {}".format(tracking_id))
# Send notification
@ -141,10 +172,8 @@ def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, kee
if not master_asset:
raise ValueError("No master asset for tracking ID: {}".format(tracking_id))
# 3. Get CreativeX score from database (lookup by original Box filename)
# The PDF contains the filename field with the full name (job + tracking ID)
# So we lookup using the original filename from Box, not the stripped version
creativex_data = db.get_creativex_score_by_filename(filename)
# 3. Get CreativeX score from database (lookup by filename, fallback to tracking ID)
creativex_data = db.get_creativex_score_by_filename(filename, tracking_id=tracking_id)
# Build box_metadata dict (for compatibility with existing code)
if creativex_data:
@ -191,7 +220,8 @@ def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, kee
parsed_filename=parsed,
box_metadata=box_metadata, # Pass CreativeX data from database
tracking_mode=tracking_mode, # Pass tracking mode for folder-only handling
master_opentext_id=master_asset['opentext_id'] # Pass master DAM ID for derivative tracking
master_opentext_id=master_asset['opentext_id'], # Primary master DAM ID
master_opentext_ids=master_opentext_ids # All master IDs (PPR: multiple, PROD: single)
)
# DRYRUN MODE: Display full asset representation and exit
@ -215,6 +245,20 @@ def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, kee
logger.info(" Score: {}".format(box_metadata.get('score')))
logger.info(" URL: {}".format(box_metadata.get('url')))
logger.info("")
# PPR ONLY: Register master asset IDs in lookup domain (even in dryrun for testing)
# This API call is safe - it only adds values to the lookup table, doesn't create assets
if master_opentext_ids:
logger.info("PPR Domain Registration Test:")
registration_result = dam.register_master_asset_ids_for_ppr(master_opentext_ids)
if registration_result.get('skipped'):
logger.info(" Skipped (not PPR environment)")
else:
logger.info(" Registered: {}".format(registration_result.get('registered_ids', [])))
if registration_result.get('failed_ids'):
logger.info(" Failed: {}".format(registration_result.get('failed_ids', [])))
logger.info("")
logger.info("DRYRUN: No upload performed, file kept in Box")
logger.info("=" * 80)
@ -248,6 +292,11 @@ def process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, kee
)
logger.info("Will upload to: 01. Final Assets/{}".format(subfolder_path))
# PPR ONLY: Register master asset IDs in lookup domain before upload
# OpenText API requires domain values to exist before they can be used in asset creation
if master_opentext_ids:
dam.register_master_asset_ids_for_ppr(master_opentext_ids)
upload_result = dam.upload_asset(
file_path=clean_temp_file,
folder_id=upload_folder_id,
@ -347,7 +396,7 @@ def main():
box = BoxClient(config, root_folder_id=config['box'].get('root_folder_a2_a3'))
db = Database(config)
notifier = Notifier(config)
parser = FilenameParser()
parser = FilenameParser(dam_base_url=dam.base_url) # Pass DAM URL for environment detection
mvp_extractor = MetadataExtractorMVP(field_mappings)
# Test connections
@ -426,7 +475,7 @@ def main():
logger.info("Processing file {}/{}".format(idx, len(valid_files)))
logger.info("=" * 60)
result = process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, keep_files=args.keep_files, dryrun=args.dryrun)
result = process_box_file(file_info, dam, box, db, parser, mvp_extractor, config, notifier, keep_files=args.keep_files, dryrun=args.dryrun)
if result['success']:
successful_files.append(result)

View file

@ -1201,6 +1201,156 @@ class DAMClient:
mime_type, _ = mimetypes.guess_type(file_path)
return mime_type or 'application/octet-stream'
def register_master_asset_id_domain_value(self, master_asset_id):
"""
Register a master asset ID in the FERRERO_MASTER_ASSET_ID lookup domain.
Required in PPR environment before using the ID in asset creation.
The OpenText API does not support creating new domain values during asset
creation, so this must be called before the create asset API.
Args:
master_asset_id: The master asset ID to register
Returns:
dict with success, http_code, and optional error
"""
# Only for PPR environment
if 'ppr' not in self.base_url.lower():
return {'success': True, 'skipped': True, 'reason': 'Not PPR environment'}
try:
payload = {
"domain_value_resource": {
"domain_value": {
"description": master_asset_id,
"display_value": master_asset_id,
"field_value": {
"type": "string",
"value": master_asset_id
}
}
}
}
logger.info("PPR: Registering master asset ID '{}' in lookup domain...".format(master_asset_id))
response = self._make_api_request(
'POST',
"{}/v6/lookupdomains/FERRERO_MASTER_ASSET_ID/lookupvalues".format(self.base_url),
json=payload,
headers={
'Content-Type': 'application/json',
'Accept': 'application/json'
}
)
# Success cases
if response.status_code in [200, 201, 202]:
logger.info("PPR: Master asset ID '{}' registered successfully".format(master_asset_id))
return {
'success': True,
'http_code': response.status_code,
'already_existed': False
}
# Already exists - OpenText returns 409 OR 500 with "duplicate code" message
if response.status_code == 409:
logger.info("PPR: Master asset ID '{}' already exists in lookup domain".format(master_asset_id))
return {
'success': True,
'http_code': response.status_code,
'already_existed': True
}
# Check for duplicate error in 500 response (OpenText quirk)
if response.status_code == 500:
try:
error_data = response.json()
error_msg = error_data.get('exception_body', {}).get('message', '')
if 'duplicate' in error_msg.lower():
logger.info("PPR: Master asset ID '{}' already exists in lookup domain".format(master_asset_id))
return {
'success': True,
'http_code': response.status_code,
'already_existed': True
}
except:
pass
# Actual failure
error_msg = "Failed to register master asset ID '{}': HTTP {} - {}".format(
master_asset_id,
response.status_code,
response.text[:200] if response.text else 'No response'
)
logger.warning(error_msg)
return {
'success': False,
'http_code': response.status_code,
'error': error_msg
}
except Exception as e:
error_msg = "Exception registering master asset ID '{}': {}".format(master_asset_id, str(e))
logger.error(error_msg)
return {
'success': False,
'error': error_msg
}
def register_master_asset_ids_for_ppr(self, master_asset_ids):
"""
Register all master asset IDs in the lookup domain (PPR only).
Call this before creating an asset that references these IDs.
The OpenText DAM API does not support creating new domain values during
asset creation. In PPR, we must first add each master asset ID to the
FERRERO_MASTER_ASSET_ID domain value table before the create asset call.
Args:
master_asset_ids: List of master asset IDs to register
Returns:
dict with success, registered_ids, failed_ids
"""
# Only for PPR environment
if 'ppr' not in self.base_url.lower():
logger.debug("Not PPR environment - skipping master asset ID domain registration")
return {'success': True, 'skipped': True}
if not master_asset_ids:
return {'success': True, 'registered_ids': [], 'failed_ids': []}
logger.info("=" * 60)
logger.info("PPR: Registering {} master asset ID(s) in lookup domain".format(len(master_asset_ids)))
logger.info(" IDs: {}".format(', '.join(master_asset_ids)))
logger.info("=" * 60)
registered = []
failed = []
for master_id in master_asset_ids:
result = self.register_master_asset_id_domain_value(master_id)
if result.get('success'):
registered.append(master_id)
else:
failed.append({'id': master_id, 'error': result.get('error')})
logger.info("PPR: Domain registration complete - {}/{} succeeded".format(
len(registered), len(master_asset_ids)))
if failed:
logger.warning("PPR: Failed to register: {}".format(
', '.join([f['id'] for f in failed])))
# Return success even if some failed (better to try the upload and see)
return {
'success': len(failed) == 0,
'registered_ids': registered,
'failed_ids': failed
}
def get_or_create_subfolder_path(self, base_folder_id, subfolder_path):
"""
Create or find subfolder structure in DAM matching Box structure

View file

@ -256,18 +256,49 @@ class Database:
# Parse JSONB as dict
full_metadata = row[3] if isinstance(row[3], dict) else json.loads(row[3])
# Parse Box info from description
box_info = self.parse_box_info_from_description(row[4])
return {
'tracking_id': row[0],
'opentext_id': row[1],
'upload_directory': row[2],
'full_metadata': full_metadata,
'description': row[4]
'description': row[4],
'box_file_id': box_info.get('box_file_id'),
'box_url': box_info.get('box_url')
}
finally:
cursor.close()
self.put_connection(conn)
@staticmethod
def parse_box_info_from_description(description):
"""
Parse Box file ID and URL from master asset description field.
Description format:
Box File ID: {id}
Box URL: {url}
DAM Asset ID: {opentext_id}
Returns:
dict with box_file_id and box_url (None if not found)
"""
result = {'box_file_id': None, 'box_url': None}
if not description:
return result
for line in description.split('\n'):
line = line.strip()
if line.startswith('Box File ID:'):
result['box_file_id'] = line.split(':', 1)[1].strip()
elif line.startswith('Box URL:'):
result['box_url'] = line.split(':', 1)[1].strip()
return result
def check_campaign_upload_complete(self, campaign_id):
"""
Check if ALL master assets for a campaign have been uploaded
@ -519,6 +550,157 @@ class Database:
cursor.close()
self.put_connection(conn)
def get_a1_retry_status(self, campaign_id):
"""
Get A1 retry status for campaign
Args:
campaign_id: DAM campaign folder ID
Returns:
dict with retry_count, last_retry_at, permanently_failed, failure_reason
Returns None if campaign not found
"""
conn = self.get_connection()
try:
cursor = conn.cursor()
cursor.execute("""
SELECT a1_retry_count, a1_last_retry_at,
a1_permanently_failed, a1_failure_reason
FROM campaign_status
WHERE campaign_id = %s
""", (campaign_id,))
row = cursor.fetchone()
if row:
return {
'retry_count': row[0] or 0,
'last_retry_at': row[1],
'permanently_failed': row[2] or False,
'failure_reason': row[3]
}
else:
return None
finally:
cursor.close()
self.put_connection(conn)
def increment_a1_retry(self, campaign_id, campaign_number, campaign_name, reason):
"""
Increment A1 retry counter and mark as permanently failed if max attempts reached
Args:
campaign_id: DAM campaign folder ID
campaign_number: Campaign number (e.g., C000000078)
campaign_name: Campaign name
reason: Description of failure (e.g., "No master assets found")
Returns:
dict with success, retry_count, permanently_failed
"""
conn = self.get_connection()
try:
cursor = conn.cursor()
# Maximum retry attempts before marking as permanently failed
MAX_RETRIES = 3
# Get current retry count
cursor.execute("""
SELECT a1_retry_count FROM campaign_status
WHERE campaign_id = %s
""", (campaign_id,))
row = cursor.fetchone()
current_count = (row[0] or 0) if row else 0
new_count = current_count + 1
is_permanently_failed = new_count >= MAX_RETRIES
# Insert or update campaign status with retry tracking
cursor.execute("""
INSERT INTO campaign_status (
campaign_id, campaign_number, campaign_name,
live_campaign, status, webhook_sent,
a1_retry_count, a1_last_retry_at,
a1_permanently_failed, a1_failure_reason
) VALUES (%s, %s, %s, 'NO', 'A1', FALSE, %s, CURRENT_TIMESTAMP, %s, %s)
ON CONFLICT (campaign_id) DO UPDATE SET
a1_retry_count = EXCLUDED.a1_retry_count,
a1_last_retry_at = EXCLUDED.a1_last_retry_at,
a1_permanently_failed = EXCLUDED.a1_permanently_failed,
a1_failure_reason = EXCLUDED.a1_failure_reason,
updated_at = CURRENT_TIMESTAMP
""", (
campaign_id,
campaign_number,
campaign_name,
new_count,
is_permanently_failed,
reason if is_permanently_failed else None
))
conn.commit()
logger.info("A1 retry tracking: Campaign {} - Attempt {}/{} (Permanently Failed: {})".format(
campaign_number, new_count, MAX_RETRIES, is_permanently_failed
))
return {
'success': True,
'retry_count': new_count,
'permanently_failed': is_permanently_failed
}
except Exception as e:
conn.rollback()
logger.error("Failed to increment A1 retry: {}".format(str(e)))
return {'success': False, 'error': str(e)}
finally:
cursor.close()
self.put_connection(conn)
def reset_a1_retry(self, campaign_id):
"""
Reset A1 retry tracking for campaign (used when campaign is fixed manually)
Args:
campaign_id: DAM campaign folder ID
Returns:
dict with success boolean
"""
conn = self.get_connection()
try:
cursor = conn.cursor()
cursor.execute("""
UPDATE campaign_status
SET a1_retry_count = 0,
a1_last_retry_at = NULL,
a1_permanently_failed = FALSE,
a1_failure_reason = NULL,
updated_at = CURRENT_TIMESTAMP
WHERE campaign_id = %s
""", (campaign_id,))
conn.commit()
logger.info("Reset A1 retry tracking for campaign: {}".format(campaign_id))
return {'success': True}
except Exception as e:
conn.rollback()
logger.error("Failed to reset A1 retry: {}".format(str(e)))
return {'success': False, 'error': str(e)}
finally:
cursor.close()
self.put_connection(conn)
def check_campaign_processed(self, campaign_id):
"""
Check if campaign has already been processed (webhook sent)
@ -693,15 +875,18 @@ class Database:
cursor.close()
self.put_connection(conn)
def get_creativex_score_by_filename(self, filename):
def get_creativex_score_by_filename(self, filename, tracking_id=None):
"""
Get CreativeX score data by filename
Performs extension-agnostic lookup: if exact filename not found,
tries common video/image extensions (.mp4, .jpg, .png, .mov, etc.)
If still not found and tracking_id provided, falls back to LIKE search
on tracking ID (handles mismatched naming from CreativeX PDFs).
Args:
filename: Filename to search for
tracking_id: Optional tracking ID for fallback lookup
Returns:
dict with creativex data or None if not found
@ -748,6 +933,24 @@ class Database:
if row:
break # Found with alternative extension
# If still not found, try tracking ID fallback
# CreativeX PDFs sometimes have different naming (extra text, stripped hyphens)
# but tracking ID is always consistent
if not row and tracking_id:
cursor.execute("""
SELECT filename, creativex_id, creativex_url, quality_score,
box_file_id, full_extraction_data, extracted_at
FROM creativex_scores
WHERE filename LIKE %s AND status = 'active'
ORDER BY extracted_at DESC
LIMIT 1
""", ('%' + tracking_id + '%',))
row = cursor.fetchone()
if row:
logger.info("CreativeX: Found score via tracking ID fallback '{}' -> {}".format(
tracking_id, row[0]))
if not row:
return None

View file

@ -15,10 +15,43 @@ class FilenameParser:
[JOB]_[BRAND]_[SUBJECT]_[ASSET]_[DUR]_[RATIO]_[SPOT]_[COUNTRY]_[LANG]_[SOCIAL]_[TRACKING]
Example: 1234567_RAF_ME-MOMENT_OLV_6S_1x1_REF_GL_it_IGF_pOiJ9s
PPR Environment: Supports multiple tracking IDs (e.g., pOiJ9s+BqB8vo+laRJo0)
PROD Environment: Single tracking ID only (backward compatible)
"""
# Known social media platform codes
SOCIAL_MEDIA_CODES = ['FBP', 'FBR', 'IGF', 'IGR'] # Expandable
# Known social media platform codes (from Ferrero naming tool data.json)
SOCIAL_MEDIA_CODES = [
# Facebook
'FBD', 'FGF', 'FBR', 'FRO', 'FBS', 'FBF', 'FBP', 'FIA', 'FIV',
'FMP', 'FPF', 'FRC', 'FSE', 'FSS', 'FSV', 'FUK', 'FVF',
# Instagram
'IGF', 'IGE', 'IGG', 'IGT', 'IPF', 'IPR', 'IGR', 'IGO', 'IGS', 'ISH', 'IST',
# Audience Network
'ANC', 'ANI', 'ANR',
# Messenger
'MSI', 'MSS',
# YouTube
'YTA', 'YTB', 'YTS',
# Other platforms
'AMZ', 'DV3', 'GOO', 'PIN', 'SNA', 'TIK', 'TWI', 'VOD',
]
def __init__(self, dam_base_url=None):
"""
Initialize parser with optional environment detection
Args:
dam_base_url: DAM base URL for environment detection (optional)
"""
self.dam_base_url = dam_base_url
self.is_ppr = self._is_ppr_environment()
def _is_ppr_environment(self):
"""Check if running in PPR environment"""
if not self.dam_base_url:
return False
return 'ppr.dam.ferrero.com' in self.dam_base_url.lower()
def parse_filename(self, filename):
"""
@ -178,21 +211,68 @@ class FilenameParser:
logger.debug("Found social media: {}".format(part))
index += 1
# Tracking ID: 6 alphanumeric, optionally with -N suffix
elif re.match(r'^[a-zA-Z0-9]{6}(-N)?$', part):
tracking = part
tracking_mode = 'full'
base_tracking_id = tracking
# Tracking ID(s): 6 alphanumeric, optionally with -N suffix
# PPR: Supports multiple IDs (e.g., "BqB8vo+SfUQ7m+laRJo0")
# PROD: Single ID only (backward compatible)
elif re.match(r'^[a-zA-Z0-9]{6}(-N)?(\+[a-zA-Z0-9]{6}(-N)?)*$', part):
# Check if multiple IDs provided
if '+' in part and self.is_ppr:
# PPR ONLY: Parse multiple tracking IDs
tracking_ids = []
tracking_modes = []
tracking_ids_with_suffix = []
if tracking.endswith('-N'):
tracking_mode = 'folder_only'
base_tracking_id = tracking[:-2] # Strip -N suffix
logger.info("Detected folder-only tracking ID: {} (base: {})".format(tracking, base_tracking_id))
id_parts = part.split('+')
logger.info("PPR Environment - Multiple tracking IDs detected: {}".format(len(id_parts)))
for tracking in id_parts:
tracking_mode = 'full'
base_tracking_id = tracking
if tracking.endswith('-N'):
tracking_mode = 'folder_only'
base_tracking_id = tracking[:-2]
logger.info("Folder-only tracking ID: {} (base: {})".format(tracking, base_tracking_id))
tracking_ids.append(base_tracking_id)
tracking_modes.append(tracking_mode)
tracking_ids_with_suffix.append(tracking)
# Store primary (first) for backward compatibility
parsed['tracking_id'] = tracking_ids[0]
parsed['tracking_mode'] = tracking_modes[0]
parsed['tracking_id_with_suffix'] = tracking_ids_with_suffix[0]
# Store all IDs for multi-master support
parsed['tracking_ids'] = tracking_ids
parsed['tracking_modes'] = tracking_modes
parsed['tracking_ids_with_suffix'] = tracking_ids_with_suffix
parsed['has_multiple_masters'] = True
logger.info("Parsed {} tracking IDs: {}".format(len(tracking_ids), ', '.join(tracking_ids)))
else:
# PROD or Single ID: Use only first tracking ID
if '+' in part:
logger.warning("PROD Environment - Multiple tracking IDs not supported, using first ID only")
part = part.split('+')[0] # Take only first ID
tracking = part
tracking_mode = 'full'
base_tracking_id = tracking
if tracking.endswith('-N'):
tracking_mode = 'folder_only'
base_tracking_id = tracking[:-2]
logger.info("Folder-only tracking ID: {} (base: {})".format(tracking, base_tracking_id))
parsed['tracking_id'] = base_tracking_id
parsed['tracking_mode'] = tracking_mode
parsed['tracking_id_with_suffix'] = tracking
parsed['tracking_ids'] = [base_tracking_id] # Single item list for compatibility
parsed['has_multiple_masters'] = False
logger.debug("Found tracking ID: {}".format(tracking))
parsed['tracking_id'] = base_tracking_id
parsed['tracking_mode'] = tracking_mode
parsed['tracking_id_with_suffix'] = tracking
logger.debug("Found tracking ID: {}".format(tracking))
index += 1
# Unknown part - could be aspect ratio fallback
@ -216,8 +296,8 @@ class FilenameParser:
def strip_upload_components(self, filename):
"""
Strip OMG Job Number and Tracking ID from filename
Returns clean filename in V2.1 order
Strip OMG Job Number from front and Tracking ID from back of filename.
Keeps everything else as-is (including social media codes, DV3, etc.)
Args:
filename: Original filename
@ -226,40 +306,23 @@ class FilenameParser:
Clean filename for upload (no job number, no tracking ID)
Example:
Input: 1234567_RAF_TEST_OLV_6S_1x1_REF_GL_it_IGF_abc123.mp4
Output: RAF_TEST_OLV_6S_1x1_REF_GL_it_IGF.mp4
Input: 6662777_NUT_XMAS-SHARETHELOVE-GLAS_OLV_6S_16X9_PL_pl_YTA_EvQJrM.mp4
Output: NUT_XMAS-SHARETHELOVE-GLAS_OLV_6S_16X9_PL_pl_YTA.mp4
"""
parsed = self.parse_filename(filename)
import os
if not parsed:
base, ext = os.path.splitext(filename)
parts = base.split('_')
if len(parts) < 3:
return filename
# Build clean filename in V2.1 order
# [BRAND]_[SUBJECT]_[ASSET]_[DUR]_[RATIO]_[SPOT]_[COUNTRY]_[LANG]_[SOCIAL]
clean_parts = []
# Strip job number from front (digits only)
if parts[0].isdigit():
parts = parts[1:]
if parsed['brand_code']:
clean_parts.append(parsed['brand_code'])
if parsed['subject_title']:
clean_parts.append(parsed['subject_title'])
if parsed['asset_type']:
clean_parts.append(parsed['asset_type'])
if parsed['seconds']:
clean_parts.append(parsed['seconds'] + 'S')
if parsed['aspect_ratio']:
clean_parts.append(parsed['aspect_ratio'])
if parsed['spot_version']:
clean_parts.append(parsed['spot_version'])
if parsed['country_code']:
clean_parts.append(parsed['country_code'])
if parsed['language_code']:
clean_parts.append(parsed['language_code'])
if parsed['social_media_version']:
clean_parts.append(parsed['social_media_version'])
# Strip tracking ID(s) from back (6 alphanumeric chars, optionally with +joined IDs or -N suffix)
if parts and re.match(r'^[a-zA-Z0-9]{6}(-N)?(\+[a-zA-Z0-9]{6}(-N)?)*$', parts[-1]):
parts = parts[:-1]
clean_filename = '_'.join(clean_parts)
if parsed['extension']:
clean_filename += parsed['extension']
return clean_filename
return '_'.join(parts) + ext

View file

@ -5,6 +5,8 @@ Compatible with Python 3.6+
"""
import logging
import json
import copy
from datetime import datetime, timedelta
import os
from shared.config_loader import load_country_code_mappings
@ -23,6 +25,7 @@ class MetadataExtractorMVP:
self.filename_updates = field_mappings.get('filename_updates', {})
self.forced_values = field_mappings.get('forced_values', {})
self.defaults = field_mappings.get('defaults', {})
self.asset_type_overrides = field_mappings.get('asset_type_overrides', {})
# Load country code mappings (ISO -> DAM codes)
self.country_mappings = load_country_code_mappings()
@ -34,6 +37,22 @@ class MetadataExtractorMVP:
if self.asset_type_mappings:
logger.info("Loaded {} asset type mappings (3-letter->DAM)".format(len(self.asset_type_mappings)))
# Load asset representation template for folder-only mode
self.template_fields = self._load_asset_representation_template()
if self.template_fields:
logger.info("Loaded asset representation template with {} fields".format(len(self.template_fields)))
def _load_asset_representation_template(self):
"""Load the asset representation template JSON for folder-only mode"""
template_path = 'config/asset_representation_template.json'
try:
with open(template_path, 'r') as f:
data = json.load(f)
return data['asset_resource']['asset']['metadata']['metadata_element_list']
except Exception as e:
logger.warning("Could not load asset representation template: {}".format(str(e)))
return []
def extract_mvp_fields(self, master_metadata):
"""
Extract only MVP fields from full master metadata
@ -94,7 +113,7 @@ class MetadataExtractorMVP:
return extracted_fields
def build_mvp_asset_representation(self, master_metadata, clean_filename, parsed_filename, box_metadata=None, tracking_mode='full', master_opentext_id=None):
def build_mvp_asset_representation(self, master_metadata, clean_filename, parsed_filename, box_metadata=None, tracking_mode='full', master_opentext_id=None, master_opentext_ids=None):
"""
Build asset representation with MVP fields + updates from filename
@ -127,15 +146,33 @@ class MetadataExtractorMVP:
mvp_fields = []
mvp_fields = self._build_fields_from_filename(parsed_filename, clean_filename)
# Apply forced values from config (e.g., AGENCY NAME)
# STATE is already handled in _build_fields_from_filename
mvp_fields = self._apply_forced_values(mvp_fields)
# Add missing MVP fields with defaults (both modes)
mvp_fields = self._add_missing_fields(mvp_fields, parsed_filename)
# Add empty required fields that DAM expects (even if empty) - folder-only mode needs these
mvp_fields = self._add_empty_required_fields(mvp_fields)
# Apply asset type overrides (e.g., EOL) - takes final precedence over forced values/defaults
mvp_fields = self._apply_asset_type_overrides(mvp_fields, parsed_filename)
# Update CreativeX fields from Box metadata if provided
if box_metadata:
mvp_fields = self._update_creativex_fields(mvp_fields, box_metadata)
# Add Master Asset ID field if provided (derivative tracking)
if master_opentext_id:
# Add MASTERASSETIDS field with all master IDs
# Priority: Use master_opentext_ids if provided (multiple IDs), otherwise fall back to single master_opentext_id
if master_opentext_ids and len(master_opentext_ids) > 0:
mvp_fields = self._add_master_asset_ids_field(mvp_fields, master_opentext_ids)
if len(master_opentext_ids) > 1:
logger.info("PPR - Added MASTERASSETIDS field with {} master IDs".format(len(master_opentext_ids)))
else:
logger.info("Added MASTERASSETIDS field with 1 master ID")
elif master_opentext_id:
# Fallback to single master ID if master_opentext_ids not provided
mvp_fields = self._add_master_asset_id_field(mvp_fields, master_opentext_id)
logger.info("Added Master Asset ID field: {}".format(master_opentext_id))
@ -190,8 +227,28 @@ class MetadataExtractorMVP:
# Update the field
for field in mvp_fields:
if field.get('id') == field_id:
self._set_field_value(field, value)
logger.info("Updated {} from filename: {}".format(field_id, value))
# For tabular fields (like MAIN_LANGUAGES), update the 'values' array
# The DAM reads from 'values' (plural), not 'value' (singular)
if field.get('type') == 'com.artesia.metadata.MetadataTableField' or 'values' in field:
field['values'] = [
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'expired_value': False,
'field_value': {
'type': 'string',
'value': value
},
'type': 'com.artesia.metadata.DomainValue'
}
}
]
logger.info("Updated tabular field {} values array from filename: {}".format(field_id, value))
else:
self._set_field_value(field, value)
logger.info("Updated {} from filename: {}".format(field_id, value))
break
# Apply country code mapping (ISO -> DAM codes)
@ -268,6 +325,88 @@ class MetadataExtractorMVP:
return mvp_fields
def _apply_asset_type_overrides(self, mvp_fields, parsed_filename):
"""
Apply asset type overrides when a matching asset type (e.g., EOL) is detected in the filename.
These overrides take final precedence over forced values and defaults.
Args:
mvp_fields: List of MVP field objects
parsed_filename: Parsed filename dict (must contain 'asset_type' key)
Returns:
Updated mvp_fields list
"""
if not parsed_filename:
return mvp_fields
asset_type = parsed_filename.get('asset_type')
if not asset_type:
return mvp_fields
overrides = self.asset_type_overrides.get(asset_type)
if not overrides:
return mvp_fields
logger.info("Applying {} asset type overrides for '{}'".format(len(overrides), asset_type))
for field_id, override_value in overrides.items():
# Empty string means remove the field entirely
if override_value == '':
before_count = len(mvp_fields)
mvp_fields = [f for f in mvp_fields if f.get('id') != field_id]
if len(mvp_fields) < before_count:
logger.info("Asset type override: removed field {}".format(field_id))
else:
logger.debug("Asset type override: field {} not present (nothing to remove)".format(field_id))
continue
field_found = False
for field in mvp_fields:
if field.get('id') == field_id:
field_found = True
# For tabular fields (like MAIN_LANGUAGES), update both 'value' and 'values'
if field.get('type') == 'com.artesia.metadata.MetadataTableField' or 'values' in field:
domain_value_obj = {
'type': 'com.artesia.metadata.DomainValue',
'field_value': {'type': 'string', 'value': override_value},
'display_value': override_value,
'expired_value': False,
'active_to': '',
'active_from': ''
}
field['value'] = {
'value': domain_value_obj,
'is_locked': False,
'domain_value': True,
'cascading_domain_value': False
}
field['values'] = [
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'expired_value': False,
'field_value': {
'type': 'string',
'value': override_value
},
'type': 'com.artesia.metadata.DomainValue'
}
}
]
logger.info("Asset type override: {} = {} (tabular)".format(field_id, override_value))
else:
self._set_field_value(field, override_value)
logger.info("Asset type override: {} = {}".format(field_id, override_value))
break
if not field_found:
logger.warning("Asset type override field '{}' not found in MVP fields - skipping".format(field_id))
return mvp_fields
def _add_missing_fields(self, mvp_fields, parsed_filename):
"""Add missing MVP fields from filename or defaults"""
field_ids = [f.get('id') for f in mvp_fields]
@ -278,29 +417,82 @@ class MetadataExtractorMVP:
language = parsed_filename['language_code'].upper()
logger.info("Adding MAIN_LANGUAGES: {}".format(language))
domain_value_obj = {
'type': 'com.artesia.metadata.DomainValue',
'field_value': {'type': 'string', 'value': language},
'display_value': language,
'expired_value': False,
'active_to': '',
'active_from': ''
}
mvp_fields.append({
'id': 'MAIN_LANGUAGES',
'name': 'MAIN LANGUAGES',
'parent_table_id': 'FERRERO.TABULAR.FIELD.MAIN LANGUAGES',
'type': 'com.artesia.metadata.MetadataTableField',
'value': {
'value': domain_value_obj,
'is_locked': False,
'domain_value': True,
'cascading_domain_value': False
},
'values': [
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'field_value': {
'type': 'string',
'value': language
},
'expired_value': False,
'field_value': {'type': 'string', 'value': language},
'type': 'com.artesia.metadata.DomainValue'
}
}
]
],
'tabular': True,
'domained': True,
'required': True,
'domain_id': 'FERRERO.DOMAIN.MAIN LAGUAGES_LU'
})
# Add other missing fields with defaults
field_ids = [f.get('id') for f in mvp_fields]
for field_id, default_value in self.defaults.items():
if field_id in field_ids:
# Field exists (e.g. from template) - check if value is empty and set default
for field in mvp_fields:
if field.get('id') == field_id:
# Tabular fields use 'values' array - skip if already populated
if field.get('type') == 'com.artesia.metadata.MetadataTableField':
if field.get('values'):
break # Already has values
# Empty tabular - fall through to add as new below
break
# Regular field - check if it has an actual value set
val = field.get('value', {})
has_value = 'value' in val and isinstance(val.get('value'), dict) and 'value' in val['value']
if not has_value:
# Use DomainValue format for domained fields
if field.get('domained', False):
field['value'] = {
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'active_from': '',
'active_to': '',
'display_value': default_value,
'expired_value': False,
'field_value': {'type': 'string', 'value': default_value},
'type': 'com.artesia.metadata.DomainValue'
}
}
else:
field['value'] = {'value': {'type': 'string', 'value': default_value}}
logger.info("Set default on template field {} = {}".format(field_id, default_value))
break
continue
if field_id not in field_ids:
logger.info("Adding {} with default: {}".format(field_id, default_value))
@ -310,12 +502,72 @@ class MetadataExtractorMVP:
]
if is_tabular:
# Map field IDs to correct parent table IDs
parent_table_map = {
'FERRERO.FIELD.ASSETCOMPLIANCE': 'FERRERO.TABULAR.FIELD.ASSETCOMPLIANCE',
'MARKETING_TAG': 'FERRERO.TABULAR.FIELD.MARKETING.TAG',
}
parent_table_id = parent_table_map.get(field_id, 'FERRERO.TABULAR.FIELD.' + field_id.split('.')[-1])
domain_value_obj = {
'type': 'com.artesia.metadata.DomainValue',
'field_value': {'type': 'string', 'value': default_value},
'display_value': default_value,
'expired_value': False,
'active_to': '',
'active_from': ''
}
mvp_fields.append({
'id': field_id,
'parent_table_id': 'FERRERO.TABULAR.FIELD.' + field_id.split('.')[-1],
'parent_table_id': parent_table_id,
'type': 'com.artesia.metadata.MetadataTableField',
'value': {
'value': domain_value_obj,
'is_locked': False,
'domain_value': True,
'cascading_domain_value': False
},
'values': [
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'expired_value': False,
'field_value': {
'type': 'string',
'value': default_value
},
'type': 'com.artesia.metadata.DomainValue'
}
}
],
'tabular': True,
'domained': True
})
else:
# Non-domain fields use simple value structure
non_domain_fields = [
'FERRERO.MARKETING.FIELD.VIDEO_POST_PROD_COMPANY',
'FERRERO.MARKETING.FIELD.AUDIO_POST_PROD_COMPANY',
]
if field_id in non_domain_fields:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'value': {
'type': 'string',
'value': default_value
}
}
})
else:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'cascading_domain_value': False,
'domain_value': True,
'value': {
@ -326,21 +578,154 @@ class MetadataExtractorMVP:
'type': 'com.artesia.metadata.DomainValue'
}
}
]
})
else:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'cascading_domain_value': False,
'domain_value': True,
'value': {
'type': 'string',
'value': default_value
})
return mvp_fields
def _apply_forced_values(self, mvp_fields):
"""
Apply forced values from config to existing fields.
For fields not yet present, adds them with DomainValue format.
Used in folder-only mode where _update_fields is not called.
"""
field_ids = [f.get('id') for f in mvp_fields]
for field_id, forced_value in self.forced_values.items():
if field_id in field_ids:
# Field exists - set value with proper format based on field type
for field in mvp_fields:
if field.get('id') == field_id:
if field.get('domained', False):
field['value'] = {
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'active_from': '',
'active_to': '',
'display_value': forced_value,
'expired_value': False,
'field_value': {'type': 'string', 'value': forced_value},
'type': 'com.artesia.metadata.DomainValue'
}
}
else:
self._set_field_value(field, forced_value)
logger.info("Forced value applied: {} = {}".format(field_id, forced_value))
break
else:
# Field not present - add with DomainValue format
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'cascading_domain_value': False,
'domain_value': True,
'value': {
'field_value': {'type': 'string', 'value': forced_value},
'type': 'com.artesia.metadata.DomainValue'
}
})
}
})
logger.info("Forced value added: {} = {}".format(field_id, forced_value))
return mvp_fields
def _add_empty_required_fields(self, mvp_fields):
"""
Add fields that the DAM expects to be present even if empty.
In full-inheritance mode these come from the master asset.
In folder-only mode they must be explicitly added.
Only adds fields not already present.
"""
field_ids = [f.get('id') for f in mvp_fields]
# Empty value structure for domained fields with no value set
empty_domained_value = {
'is_locked': False,
'domain_value': False,
'cascading_domain_value': False
}
# Fields with empty domained values
empty_domained_fields = [
'FERRERO.FIELD.MARKETING.FLAVOUR',
'FERRERO.FIELD.MARKETING.SIZE',
'FERRERO.FIELD.SUB BRAND',
'FERRERO.MARKET.FIELD.BUYOUT',
'FERRERO.MARKET.FIELD.FERRERO PROPERTY',
'FERRERO.MARKET.VID_N_STAT',
'FERRERO.MARKETING.FIELD.SPOT_VERSION',
]
for field_id in empty_domained_fields:
if field_id not in field_ids:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': dict(empty_domained_value)
})
# Fields with empty non-domained values
empty_plain_fields = [
'FERRERO.MARKETING.FIELD.DIRECTOR_NAME',
'FERRERO.MARKETING.FIELD.VID_POST_PROD_CONTACT',
'FERRERO.MARKETING.FIELD.AUDIO_POST_PROD_CONTACT',
'FERRERO.MARKET.FIELD.LICENSE',
]
for field_id in empty_plain_fields:
if field_id not in field_ids:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'is_locked': False,
'domain_value': False,
'cascading_domain_value': False
}
})
# Domained fields with default "No" value
no_value_fields = [
'FERRERO.MARKET.FIELD.IPRIGHT',
'FERRERO.MARKET.FIELD.LICENSIN',
]
for field_id in no_value_fields:
if field_id not in field_ids:
mvp_fields.append({
'id': field_id,
'type': 'com.artesia.metadata.MetadataField',
'value': {
'value': {
'type': 'com.artesia.metadata.DomainValue',
'field_value': {'type': 'string', 'value': 'No'},
'display_value': 'No',
'expired_value': False,
'active_to': '',
'active_from': ''
},
'is_locked': False,
'domain_value': True,
'cascading_domain_value': False
}
})
# Empty tabular field: Type of Video & Static Right
if 'FERRERO.MARKET.FIELD.TYPE_VID' not in field_ids:
mvp_fields.append({
'id': 'FERRERO.MARKET.FIELD.TYPE_VID',
'parent_table_id': 'FERRERO.TABULAR.VID_STAT_TYPE',
'type': 'com.artesia.metadata.MetadataTableField',
'values': [],
'tabular': True,
'domained': True
})
added_count = len(mvp_fields) - len(field_ids)
if added_count > 0:
logger.info("Added {} empty required fields for DAM compatibility".format(added_count))
return mvp_fields
@ -415,63 +800,87 @@ class MetadataExtractorMVP:
def _build_fields_from_filename(self, parsed_filename, clean_filename):
"""
Build ALL metadata fields from parsed filename
Used in folder-only mode (tracking ID with -N suffix)
Build ALL metadata fields from parsed filename using the reference template.
Used in folder-only mode (tracking ID with -N suffix).
Note: Uses codes directly for now. Can add lookup tables later
for brand_code->brand_name, country_code->country_name, etc.
Deep copies the asset representation template and populates values
from the parsed filename. This ensures all fields have the full metadata
structure (column_name, data_type, etc.) that the DAM API requires.
"""
fields = []
if not self.template_fields:
logger.error("No asset representation template loaded - folder-only mode cannot proceed")
return []
# Deep copy the template so we don't modify the original
fields = copy.deepcopy(self.template_fields)
# Build lookup for quick access
fields_by_id = {f['id']: f for f in fields}
# Helper to set a domained field value with DomainValue structure
def set_domained_value(field, value):
field['value'] = {
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'active_from': '',
'active_to': '',
'display_value': value,
'expired_value': False,
'field_value': {'type': 'string', 'value': value},
'type': 'com.artesia.metadata.DomainValue'
}
}
# Helper to set a plain string field value
def set_string_value(field, value):
field['value'] = {'value': {'type': 'string', 'value': value}}
# --- Populate fields from filename ---
# ASSET NAME
fields.append({
'id': 'ARTESIA.FIELD.ASSET NAME',
'value': {'value': {'value': clean_filename}}
})
if 'ARTESIA.FIELD.ASSET NAME' in fields_by_id:
set_string_value(fields_by_id['ARTESIA.FIELD.ASSET NAME'], clean_filename)
# DESCRIPTION (from subject_title)
if parsed_filename.get('subject_title'):
fields.append({
'id': 'ARTESIA.FIELD.ASSET DESCRIPTION',
'value': {'value': {'value': parsed_filename['subject_title']}}
})
# DESCRIPTION
if parsed_filename.get('subject_title') and 'ARTESIA.FIELD.ASSET DESCRIPTION' in fields_by_id:
set_string_value(fields_by_id['ARTESIA.FIELD.ASSET DESCRIPTION'], parsed_filename['subject_title'])
# BRAND (use code for now, could add lookup later)
if parsed_filename.get('brand_code'):
fields.append({
'id': 'FERRERO.FIELD.BRAND',
'value': {'value': {'value': parsed_filename['brand_code']}}
})
# Note: BRAND and COUNTRY are NOT set in the metadata payload.
# They are inherited from the DAM folder structure.
# COUNTRY (map ISO code to DAM code)
if parsed_filename.get('country_code'):
dam_country_code = self._map_country_code(parsed_filename['country_code'])
fields.append({
'id': 'FERRERO.FIELD.COUNTRY',
'value': {'value': {'value': dam_country_code}}
})
# LANGUAGE (use code for now)
if parsed_filename.get('language_code'):
fields.append({
'id': 'FERRERO.FIELD.LANGUAGES',
'value': {'value': {'value': parsed_filename['language_code']}}
})
# ASSET TYPE (use code for now)
# ASSET TYPE (use config field ID, map code via lookup)
if parsed_filename.get('asset_type'):
fields.append({
'id': 'FERRERO.FIELD.ASSET TYPE',
'value': {'value': {'value': parsed_filename['asset_type']}}
})
asset_type_field_id = 'FERRERO.FIELD.ASSET TYPE'
for field_id, config in self.filename_updates.items():
if config.get('source') == 'asset_type':
asset_type_field_id = field_id
break
# STATE (force to Local)
fields.append({
'id': 'FERRERO.FIELD.STATE',
'value': {'value': {'value': 'Local'}}
})
mapped_asset_type = self._map_asset_type(parsed_filename['asset_type'])
if asset_type_field_id in fields_by_id:
set_domained_value(fields_by_id[asset_type_field_id], mapped_asset_type)
logger.info("Built {} fields from filename (folder-only mode)".format(len(fields)))
# STATE (forced to Local)
if 'FERRERO.FIELD.STATE' in fields_by_id:
set_domained_value(fields_by_id['FERRERO.FIELD.STATE'], 'Local')
# VALIDITY DATES (Start = Today, End = Today + 1 Year)
try:
today = datetime.now()
one_year_later = today + timedelta(days=365)
start_date_str = today.strftime('%m/%d/%Y')
end_date_str = one_year_later.strftime('%m/%d/%Y')
if 'FERRERO.FIELD.ASSET VALIDITY START PERIOD' in fields_by_id:
set_string_value(fields_by_id['FERRERO.FIELD.ASSET VALIDITY START PERIOD'], start_date_str)
if 'FERRERO.FIELD.ASSET VALIDITY END PERIOD' in fields_by_id:
set_string_value(fields_by_id['FERRERO.FIELD.ASSET VALIDITY END PERIOD'], end_date_str)
except Exception as e:
logger.error("Failed to set validity dates in folder-only mode: {}".format(str(e)))
logger.info("Built {} fields from template (folder-only mode)".format(len(fields)))
return fields
@ -757,7 +1166,56 @@ class MetadataExtractorMVP:
}
})
logger.info("Added new Master Asset ID field: {}".format(master_field_id))
return mvp_fields
def _add_master_asset_ids_field(self, mvp_fields, master_opentext_ids):
"""
Add FERRERO.MASTERASSETIDS tabular field with multiple master asset IDs
Supports Many-to-Many relationship between derivatives and masters
Args:
mvp_fields: List of MVP fields
master_opentext_ids: List of DAM Asset IDs of master assets
Returns:
Updated mvp_fields list with FERRERO.MASTERASSETIDS
"""
if not master_opentext_ids or len(master_opentext_ids) == 0:
logger.info("No master_opentext_ids provided - skipping FERRERO.MASTERASSETIDS field")
return mvp_fields
# Check if field already exists
for field in mvp_fields:
if self._get_field_id(field) == 'FERRERO.MASTERASSETIDS':
logger.info("FERRERO.MASTERASSETIDS already present - skipping")
return mvp_fields
# Build values array with all master asset IDs
values = []
for master_id in master_opentext_ids:
values.append({
'cascading_domain_value': False,
'domain_value': False,
'is_locked': False,
'value': {
'type': 'string',
'value': master_id
}
})
# Create tabular field
new_field = {
'id': 'FERRERO.MASTERASSETIDS',
'parent_table_id': 'FERRERO.TABULAR.FIELD.MASTERASSETIDS',
'type': 'com.artesia.metadata.MetadataTableField',
'values': values
}
mvp_fields.append(new_field)
logger.info("Added FERRERO.MASTERASSETIDS field with {} master asset ID(s): {}".format(
len(values), ', '.join(master_opentext_ids[:3]) + ('...' if len(master_opentext_ids) > 3 else '')))
return mvp_fields
def _get_field_id(self, field):

View file

@ -203,8 +203,28 @@ class MetadataExtractorMVP:
# Update the field
for field in mvp_fields:
if field.get('id') == field_id:
self._set_field_value(field, value)
logger.info("Updated {} from filename: {}".format(field_id, value))
# For tabular fields (like MAIN_LANGUAGES), update the 'values' array
# The DAM reads from 'values' (plural), not 'value' (singular)
if field.get('type') == 'com.artesia.metadata.MetadataTableField' or 'values' in field:
field['values'] = [
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'expired_value': False,
'field_value': {
'type': 'string',
'value': value
},
'type': 'com.artesia.metadata.DomainValue'
}
}
]
logger.info("Updated tabular field {} values array from filename: {}".format(field_id, value))
else:
self._set_field_value(field, value)
logger.info("Updated {} from filename: {}".format(field_id, value))
break
# Apply country code mapping (ISO -> DAM codes)

View file

@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""
Test script to verify MASTERASSETIDS field implementation
Shows master assets and their potential derivatives
"""
import os
import sys
import psycopg2
from dotenv import load_dotenv
# Load env vars from current directory
script_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(os.path.join(script_dir, '.env'))
try:
conn = psycopg2.connect(
host=os.getenv('DB_HOST', 'localhost'),
port=os.getenv('DB_PORT', '5437'),
database='ferrero_tracking',
user=os.getenv('DB_USER'),
password=os.getenv('DB_PASSWORD')
)
cursor = conn.cursor()
print("=" * 80)
print("MASTERASSETIDS FIELD TESTING REPORT")
print("=" * 80)
# 1. Show master assets available for testing
print("\n📋 MASTER ASSETS (Available for Testing)")
print("-" * 80)
cursor.execute("""
SELECT
tracking_id,
opentext_id,
local_campaign_id,
original_filename,
created_at
FROM master_assets
ORDER BY created_at DESC
LIMIT 10
""")
print(f"{'Tracking ID':<12} {'OpenText ID':<45} {'Campaign':<15} {'Filename':<30}")
print("-" * 80)
for row in cursor.fetchall():
tracking_id, opentext_id, campaign_id, filename, created_at = row
filename_short = (filename[:27] + '...') if filename and len(filename) > 30 else filename or 'N/A'
print(f"{tracking_id:<12} {opentext_id:<45} {campaign_id:<15} {filename_short:<30}")
# 2. Show derivative assets (if any exist)
print("\n\n📦 DERIVATIVE ASSETS (Uploaded from Agency)")
print("-" * 80)
cursor.execute("""
SELECT
da.tracking_id,
da.dam_asset_id,
da.derivative_filename,
ma.opentext_id as master_opentext_id,
ma.local_campaign_id,
da.created_at
FROM derivative_assets da
LEFT JOIN master_assets ma ON da.tracking_id = ma.tracking_id
ORDER BY da.created_at DESC
LIMIT 10
""")
derivative_rows = cursor.fetchall()
if derivative_rows:
print(f"{'Tracking ID':<12} {'Derivative DAM ID':<45} {'Master DAM ID (should be in MASTERASSETIDS)':<50}")
print("-" * 80)
for row in derivative_rows:
tracking_id, dam_asset_id, filename, master_opentext_id, campaign_id, created_at = row
print(f"{tracking_id:<12} {dam_asset_id or 'N/A':<45} {master_opentext_id or 'N/A':<50}")
else:
print("(No derivative assets found)")
print("\n Derivatives are created when Agency returns localized assets (A2→A3 flow)")
# 3. Show campaigns ready for testing
print("\n\n🧪 CAMPAIGNS READY FOR TESTING")
print("-" * 80)
cursor.execute("""
SELECT
cs.campaign_number,
cs.campaign_name,
cs.status,
COUNT(ma.id) as master_count,
MAX(cs.updated_at) as last_updated
FROM campaign_status cs
LEFT JOIN master_assets ma ON cs.campaign_number = ma.local_campaign_id
WHERE cs.status IN ('A2', 'A3')
GROUP BY cs.campaign_number, cs.campaign_name, cs.status
ORDER BY last_updated DESC
""")
test_campaigns = cursor.fetchall()
if test_campaigns:
print(f"{'Campaign':<15} {'Status':<8} {'Master Assets':<15} {'Campaign Name':<40}")
print("-" * 80)
for row in test_campaigns:
campaign_num, campaign_name, status, count, last_updated = row
print(f"{campaign_num:<15} {status:<8} {count:<15} {campaign_name[:37]}")
else:
print("(No campaigns in A2 or A3 status)")
# 4. Get a sample tracking ID for testing
print("\n\n🔬 TEST SCENARIO")
print("-" * 80)
cursor.execute("""
SELECT tracking_id, opentext_id, local_campaign_id, original_filename
FROM master_assets
ORDER BY created_at DESC
LIMIT 1
""")
sample = cursor.fetchone()
if sample:
tracking_id, opentext_id, campaign_id, filename = sample
print(f"Sample Master Asset for Testing:")
print(f" Tracking ID: {tracking_id}")
print(f" OpenText ID: {opentext_id}")
print(f" Campaign: {campaign_id}")
print(f" Filename: {filename or 'N/A'}")
print(f"\nTo test MASTERASSETIDS field:")
print(f" 1. Upload a derivative file to Box with tracking ID: {tracking_id}")
print(f" 2. Run: python scripts/a2_to_a3_upload_polling.py --dryrun")
print(f" 3. Check for FERRERO.MASTERASSETIDS field with value: {opentext_id}")
print(f"\nNote: Field is only active in PPR environment (ppr.dam.ferrero.com)")
# 5. Environment check
print("\n\n🌍 ENVIRONMENT CONFIGURATION")
print("-" * 80)
dam_url = os.getenv('DAM_BASE_URL', 'Not configured')
print(f"DAM Base URL: {dam_url}")
if 'ppr.dam.ferrero.com' in dam_url:
print("Environment: PPR (MASTERASSETIDS field is ENABLED ✅)")
elif 'dam.ferrero.com' in dam_url:
print("Environment: PROD (MASTERASSETIDS field is DISABLED ⚠️)")
else:
print("Environment: Unknown")
print("\n" + "=" * 80)
conn.close()
except Exception as e:
print(f"❌ Error: {e}")
sys.exit(1)

View file

@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""
Test script to demonstrate MASTERASSETIDS field with multiple master asset IDs
This creates a test JSON structure showing how multiple master assets would be linked
"""
import json
# Example: A localized asset (derivative) that references TWO master assets
# Master 1: fc5c389776516bb58044c7d4bf479da458599baf (tracking: BqB8vo)
# Master 2: ad3948d72ea8550a338a600ae87a1bdd1968b066 (tracking: SfUQ7m)
test_field_structure = {
'id': 'FERRERO.MASTERASSETIDS',
'parent_table_id': 'FERRERO.TABULAR.FIELD.MASTERASSETIDS',
'type': 'com.artesia.metadata.MetadataTableField',
'values': [
# First master asset ID
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'field_value': {
'type': 'string',
'value': 'fc5c389776516bb58044c7d4bf479da458599baf'
},
'type': 'com.artesia.metadata.DomainValue'
}
},
# Second master asset ID
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'field_value': {
'type': 'string',
'value': 'ad3948d72ea8550a338a600ae87a1bdd1968b066'
},
'type': 'com.artesia.metadata.DomainValue'
}
},
# Third master asset ID (optional)
{
'cascading_domain_value': False,
'domain_value': True,
'is_locked': False,
'value': {
'field_value': {
'type': 'string',
'value': '020d76f957ec9f4ec0b18035a2d012cd3fd376c2'
},
'type': 'com.artesia.metadata.DomainValue'
}
}
]
}
print("=" * 80)
print("MULTIPLE MASTER ASSET IDS - TEST STRUCTURE")
print("=" * 80)
print()
print("Field ID:", test_field_structure['id'])
print("Parent Table:", test_field_structure['parent_table_id'])
print("Number of Master Asset IDs:", len(test_field_structure['values']))
print()
print("Master Asset IDs:")
for i, value_obj in enumerate(test_field_structure['values'], 1):
master_id = value_obj['value']['field_value']['value']
print(f" {i}. {master_id}")
print()
print("Full JSON Structure:")
print("-" * 80)
print(json.dumps(test_field_structure, indent=2))
print()
print("=" * 80)
print("TESTING NOTES")
print("=" * 80)
print()
print("To test if DAM accepts multiple IDs:")
print("1. Check if FERRERO.TABULAR.FIELD.MASTERASSETIDS schema allows multiple rows")
print("2. Verify with DAM admin if field has 'Allow Multiple Values' enabled")
print("3. Test upload with this structure to PPR environment")
print()
print("Current Implementation:")
print(" - Code adds ONE master ID (from tracking ID lookup)")
print(" - Supports Many-to-Many relationship conceptually")
print(" - Array structure ready for multiple values")
print()
print("To enable multiple IDs in production:")
print(" - Agency tool needs to send list of master tracking IDs")
print(" - Database schema needs multiple master references")
print(" - Code modification needed to look up multiple masters")
print()