Commit graph

28 commits

Author SHA1 Message Date
nickviljoen
6d6213024a Fix: Merge A+B live campaigns into single CSV for OMG
OMG's Box automation treats each new live_campaigns_*.csv as a full-list
replacement, so the per-series global CSV introduced 2026-04-30 stomped
the local list whenever a B1→B2 ran. Collapse to one combined CSV
(A-series + B-series) emitted by every handler.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 17:36:43 +02:00
nickviljoen
ba4f1a9bf7 Feature: Global live campaigns CSV + B4 closure flow
Wires B-series (global) campaigns into OMG using the same Box
automation as A-series. Mirrors the A1/A4 lifecycle for B1/B4.

- b1_to_b2_download: after B2 status update, mark live=YES status=B2
  and upload live_campaigns_global_<ts>.csv to the existing Box folder
  (BOX_LIVE_CAMPAIGNS_FOLDER_ID, 352181382858 in PROD). Filename keeps
  the live_campaigns_ prefix so the existing OMG automation rule picks
  it up.
- b4_box_uploader (new): polls DAM for status B4, marks live=NO, regens
  the global CSV. Mirrors a4_box_uploader.
- a4_box_uploader: reads prior status before overwriting; if it was
  B-series, regenerate the global CSV instead. b4_box_uploader does the
  symmetric A-series fallback. Defensive in case DAM doesn't enforce
  type-specific status transitions.
- database: add get_all_live_global_campaigns() (status LIKE 'B%').
  Tighten get_all_live_campaigns() to status LIKE 'A%' so any cross-type
  rows can't leak into the wrong CSV.
- orchestrator + orchestrator-prod: register B4 Box Uploader at 10min.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 18:12:49 +02:00
nickviljoen
5909e017a4 Reporting: Format CreativeX score as '100 (DV360)' in B1→B2 emails
DAM stores the CreativeX tabular cell as '<platform>^<score>', e.g.
'DV360^100'. Add format_cx_score_for_display() and apply at the point
where the email asset dict is built — both new-download and skipped
paths. Raw value stays in creativex_scores.quality_score so all platform
info is preserved for queries; only the email display is reshaped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 12:04:27 +02:00
nickviljoen
8bf8dc1325 Fix: Recursively walk metadata_element_list when extracting CreativeX
Diagnostic confirmed FERRERO.TAB.FIELD.CREATIVEX (score) lives at depth 2
in B1 master metadata — nested under FERRERO.TABULAR.FIELD.CREATIVEX
inside a category — and FERRERO.FIELD.CREATIVEX LINK lives at depth 1.
The flat top-level walk used previously never reached them, so live B1
runs and the backfill both reported zero CX scores. Updated extractor
in b1_to_b2_download.py and the inline copy in
backfill_b1_creativex_scores.py to descend recursively.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 11:53:15 +02:00
nickviljoen
f28b5221f7 Enhancement: Capture CreativeX score on B1→B2 global masters
Extracts CreativeX score and URL from DAM master metadata during the
B1→B2 download, persists to creativex_scores with new status
'b1-master-cx-score' (dedup by tracking_id), and surfaces the score in
the b1_to_b2_complete and b1_to_b2_partial emails — falling back to
"No CreativeX Score" when the master has no score yet. Skipped
already-downloaded assets backfill from full_metadata JSONB on next pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 11:31:07 +02:00
nickviljoen
2c06f3936f Reporting: Split new vs previously-downloaded assets in A1→A2 / B1→B2 emails
When a campaign is re-opened (status reset to A1/B1 after new files are
added), the tool correctly skips already-downloaded assets but the email
report and CSV previously listed the whole folder as "processed", which
was misleading. Reports now show "Total: 14 (12 previously downloaded,
2 new this run)" with new assets in full detail and previously-downloaded
assets in a compact list. B1→B2 CSV gains a Status column matching A1→A2.
2026-04-23 14:11:00 +02:00
nickviljoen
0408d282a5 Revert "Fix: Skip subfolders with numeric extensions in B1→B2 downloads"
This reverts commit 4dff200e10.
2026-04-10 09:44:41 +02:00
nickviljoen
4dff200e10 Fix: Skip subfolders with numeric extensions in B1→B2 downloads
DAM subfolder "WND_PCS 2026 2.0" was being treated as a downloadable
asset because ".0" passed the existing extension check. Added safeguard
to skip items with numeric-only extensions (e.g. .0, .1) which are
version numbers in folder names, not real files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 09:42:29 +02:00
nickviljoen
39a495e4cc Fix: Skip already-processed assets on B1→B2 retry runs
Previously the script re-downloaded and re-uploaded all assets on every
retry, even those already successfully stored in DB and Box. For large
campaigns (1300+ assets) this caused unnecessary load and duplicate uploads.

Now checks DB via find_global_master_by_opentext_id() before downloading.
Assets already in DB with a valid Box URL are skipped and counted toward
the processed total, so only genuinely failed assets are retried.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 09:07:07 +02:00
DJP
a7d17dd00d Add CSV attachment with campaign details to A1 and B1 success emails 2025-12-07 19:27:36 -05:00
DJP
160ef8ad43 Implement prefix-based tracking ID system for master files
- Master files (B1→B2) now always start with 'M' prefix
- Regular files (A1→A2, A5→A6) never start with 'M'
- Updated generate_unique_tracking_id() to accept is_master parameter
- All tracking IDs remain 6 characters in length
- No database schema changes required
2025-12-06 10:01:05 -05:00
DJP
5f389f6490 Implement robust Box name sanitization in shared/common.py 2025-12-01 15:23:01 -05:00
DJP
2a8129f428 Fix Box folder naming: sanitize forward slashes in campaign names 2025-12-01 15:13:58 -05:00
DJP
67c8765512 Add fallback check to B1->B2 script: skip assets with no file extension 2025-12-01 14:22:59 -05:00
DJP
0f6e816255 Fix recursive download in B1->B2 script: improve folder identification and add safeguards 2025-12-01 14:17:14 -05:00
DJP
07bce09d65 Fix B1→B2 bug: total_assets referenced before assignment 2025-11-26 15:07:27 -05:00
DJP
6fe2ba234b Implement Auth V2 (Hybrid mTLS/OAuth) and update field mappings 2025-11-21 16:46:37 -05:00
DJP
8e7ae7e2d2 Add optional mTLS certificate authentication with --auth-pfx flag
Implements dual authentication system: OAuth2 (default) + mTLS (opt-in).
Zero-risk implementation - existing OAuth2 workflows unchanged.

NEW FEATURE: mTLS Certificate Authentication
- PFX/P12 certificate support for enhanced security
- Activated ONLY with --auth-pfx command-line flag
- OAuth2 remains default (no flag = OAuth2 as before)
- Perfect for testing new auth without breaking production

USAGE:
  Default (OAuth2):
    python scripts/a1_to_a2_download.py

  With mTLS:
    python scripts/a1_to_a2_download.py --auth-pfx

IMPLEMENTATION:

1. Certificate Storage (SECURE):
   - NEW: config/certificates/ folder (gitignored)
   - Moved PFX file to secure location
   - File permissions: 600 (owner read/write only)
   - Password stored in .env (already gitignored)

2. Configuration:
   - .env: Added DAM_MTLS_CERT_PATH and DAM_MTLS_CERT_PASSWORD
   - config.yaml: Added mtls_cert_path and mtls_cert_password
   - .gitignore: Added config/certificates/, *.pfx, *.p12

3. DAM Client Dual Auth:
   - NEW: pfx_to_pem() - Converts PFX to temporary PEM for requests
   - UPDATED: __init__() - Accepts use_mtls flag
   - NEW: _make_api_request() - Unified request wrapper
   - Auto-selects auth method based on flag
   - Updated ALL 8 API calls to use wrapper

4. Scripts Updated (argparse):
   - test_connection.py - Added --auth-pfx flag
   - a1_to_a2_download.py - Added --auth-pfx flag
   - a5_to_a6_download.py - Added --auth-pfx flag
   - b1_to_b2_download.py - Added --auth-pfx flag

5. Test Script:
   - NEW: test_mtls_cert.py - Standalone cert loading test
   - Tests PFX→PEM conversion without API calls
   - Verifies certificate format and cleanup

TESTING RESULTS:
✓ Certificate loads successfully (10930 bytes)
✓ PFX→PEM conversion works (13520 bytes)
✓ Temp file cleanup working
✓ OAuth2 connection test: PASS
✓ mTLS connection test: PASS
✓ Both auth methods working independently

SECURITY:
✓ Certificate file gitignored
✓ Password in .env (gitignored)
✓ File permissions: 600
✓ Temp PEM files auto-deleted
✓ No secrets in code or config

MIGRATION PATH:
- Dev: Use dam-mtls-dev.pfx (current)
- Prod: Replace cert file, update password, same code

BACKWARD COMPATIBILITY:
✓ OAuth2 still default (100% backward compatible)
✓ Existing cron jobs unchanged
✓ No breaking changes
✓ Easy rollback (just don't use --auth-pfx)

Changes:
- .gitignore (+3 lines)
- Python-Version/.env (+3 lines)
- Python-Version/config/config.yaml (+3 lines)
- Python-Version/scripts/shared/dam_client.py (+100 lines dual auth)
- Python-Version/scripts/a1_to_a2_download.py (+14 lines argparse)
- Python-Version/scripts/a5_to_a6_download.py (+14 lines argparse)
- Python-Version/scripts/b1_to_b2_download.py (+14 lines argparse)
- Python-Version/scripts/test_connection.py (+15 lines argparse)
- NEW: Python-Version/scripts/test_mtls_cert.py (92 lines)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 18:01:23 -05:00
DJP
1081cfd9ca Add 'no assets found' email notifications for A1→A2 and B1→B2
Ensures all workflows send email notifications even when no assets are found.

NEW EMAIL TEMPLATES:
1. a1_to_a2_no_assets - Warns when A1 campaign has no master assets
2. b1_to_b2_no_assets - Warns when B1 campaign has no global master assets

Both templates use orange warning theme and explain:
- Campaign is set to A1/B1 but no assets found
- Status NOT updated (remains at current status)
- Script will retry on next run
- Suggests verifying assets exist in folder

Changes:
- a1_to_a2_download.py: Send email when total_assets == 0
- b1_to_b2_download.py: Send email when total_assets == 0
- notifier.py: Add 2 new warning templates

This completes email coverage for all scenarios:
✓ Success (all assets processed)
✓ Partial (some failed)
✓ No assets found (campaign empty)
✓ No rejections (A5→A6 specific)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 15:33:25 -05:00
DJP
055fc9ad16 Add recursive folder search, NOT APPROVED filtering, and rejection details for A5→A6
Major enhancements to all workflow scripts with recursive search and detailed rejection tracking.

NEW FEATURES:
1. Recursive Folder Search (ALL workflows: A1→A2, A5→A6, B1→B2)
   - Searches subfolders within Master/Final Assets folders
   - Preserves folder structure in Box
   - Adds 'folder_path' attribute to each asset

2. NOT APPROVED Filtering (A5→A6 ONLY)
   - Only downloads assets with ECOMMERCE STATUS = "NOT APPROVED"
   - Skips approved/other status assets
   - Logs rejected vs skipped counts

3. Rejection Details Extraction (A5→A6)
   - Extracts comments from 3 reviewers: Approver, Legal, IA&CC
   - Includes certifier names and dates
   - Displays in detailed email notifications

CHANGES BY FILE:

dam_client.py:
- NEW: _get_assets_recursive() - Recursively searches folders
- UPDATED: get_master_assets() - Now uses recursive search, adds folder_path to assets
- NEW: is_asset_not_approved() - Checks FERRERO.FIELD.ECOMMERCE STATUS
- NEW: extract_rejection_details() - Extracts all rejection comments from 10 fields

box_client.py:
- UPDATED: upload_with_tracking_id() - Added subfolder_path parameter
- NEW: _get_or_create_subfolder_path() - Creates/navigates Box subfolders
- Preserves DAM folder structure in Box uploads

a1_to_a2_download.py:
- Added folder_path extraction from assets
- Pass subfolder_path to Box upload
- Logs subfolder info during processing

b1_to_b2_download.py:
- Added folder_path extraction from assets
- Pass subfolder_path to Box upload
- Logs subfolder info during processing

a5_to_a6_download.py:
- Filter assets for NOT APPROVED status ONLY
- Extract rejection details for each asset
- Pass subfolder_path to Box upload
- Updated email data with rejection_details
- Handle "no rejections" scenario with email
- Updated logging to show rejected vs skipped counts

notifier.py:
- REPLACED: a5_to_a6_complete → a5_to_a6_rejections
- Detailed HTML template with rejection sections
- Shows Approver, Legal, and IA&CC rejections
- Styled with red warnings and bordered sections
- NEW: a5_to_a6_no_rejections template
- Green success message when no rejected assets found
- UPDATED: a5_to_a6_partial - Now uses rejected_assets

FIELD IDs EXTRACTED (A5→A6):
- FERRERO.FIELD.ECOMMERCE STATUS (primary check)
- FERRERO.MARKETING.FIELD.CERTIFIER COMMENT
- FERRERO.FIELD.ECOMMERCE CERTIFIER
- FERRERO.MARKETING.FIELD.APPROVAL DATE
- FERRERO.MARKETING.FIELD.LEGAL COMMENT
- FERRERO.FIELD.LEGAL CERTIFER (typo in field ID)
- FERRERO.MARKETING.FIELD.LEGAL APPROVAL DATE
- FERRERO.MARKETING.FIELD.IA CC COMMENT
- FERRERO.MARKETING.FIELD.IA CERTIFIER
- FERRERO.MARKETING.FIELD.IA CC APPROVAL DATE

TESTING:
✓ All connections working (DAM, Box, Database)
✓ A5→A6 script executes correctly
✓ Recursive search working
✓ NOT APPROVED filtering working
✓ "No rejections" email sent successfully
✓ Folder structure preserved in logs

WORKFLOW IMPACTS:
- A1→A2: Now searches recursively, preserves folder structure
- A5→A6: Filters for NOT APPROVED only, shows rejection details
- B1→B2: Now searches recursively, preserves folder structure

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 14:49:19 -05:00
DJP
3c5bcbf03a Fix B1→B2 to include campaign number in Box folder name
Issue: Folder named 'MASTERS_NUTELLA PLANT-BASED LAUNCH-'
Missing: Campaign number (C000000068)

Fix: Include campaign number in folder name
- campaign_id: MASTERS_C000000068
- campaign_name: NUTELLA_PLANT_BASED_LAUNCH
- Result: MASTERS_C000000068-NUTELLA_PLANT_BASED_LAUNCH

Format:
MASTERS_[CampaignNumber]-[CampaignName]

Examples:
- MASTERS_C000000068-NUTELLA_PLANT_BASED_LAUNCH
- MASTERS_C000000069-KINDER_SURPRISE_GLOBAL

Spaces and dashes in campaign name converted to underscores.

🤖 Generated with Claude Code
2025-11-04 09:40:58 -05:00
DJP
29308bdfc0 Fix B1→B2 Box folder naming - Use MASTERS-Campaign_Name format
Issue: Folder named 'MASTERS_NUTELLA PLANT-BASED LAUNCH-' (with trailing dash)
Cause: Passing entire folder name as campaign_id, empty campaign_name

Fix: Use BoxClient's built-in naming logic
- campaign_id: 'MASTERS' (prefix)
- campaign_name: 'NUTELLA_PLANT_BASED_LAUNCH' (cleaned)
- Result: MASTERS-NUTELLA_PLANT_BASED_LAUNCH

Changes:
- Replace spaces with underscores
- Replace dashes with underscores
- BoxClient adds dash between ID and name
- Final format: MASTERS-Campaign_Name

Example Results:
Before: MASTERS_NUTELLA PLANT-BASED LAUNCH-
After:  MASTERS-NUTELLA_PLANT_BASED_LAUNCH

Clean, consistent folder naming for Global Masters!

🤖 Generated with Claude Code
2025-11-04 09:36:08 -05:00
DJP
3e21b90f6a Add B1→B2 email templates and remove webhook from B1→B2 workflow
Email Templates Added:

1. b1_to_b2_complete
   - Subject: Global Master Assets Downloaded
   - Shows campaign name, ID, asset count
   - Lists all processed assets with tracking IDs and Box URLs
   - Notes Box folder: 349261192115
   - Status updated: B1 → B2

2. b1_to_b2_partial
   - Subject: Partial Download - Global Campaign
   - Shows successful and failed assets separately
   - Each asset listed with name, tracking ID, error
   - Notes status NOT updated (remains B1)
   - Mentions automatic retry

Webhook Removed from B1→B2:
- B1→B2 workflow now only sends email (no webhook)
- Webhook only for A1→A2 workflow
- Simplified B1→B2 notifications

Email will now render properly with formatted HTML instead of raw dict.

Next B1→B2 run will send properly formatted email!

🤖 Generated with Claude Code
2025-11-03 14:04:14 -05:00
DJP
9b42b6206c Fix final B1→B2 log message in Python script
Changed: 'All assets processed - Updating status A1 → A2'
To: 'All assets processed - Updating status B1 → B2'

All log messages now correctly reference B1→B2 workflow.

Test output should now show:
- Searching for B1 Global campaigns
- All assets processed - Updating status B1 → B2
- ✓ Status updated: B1 → B2

Complete and correct! 🎊

🤖 Generated with Claude Code
2025-11-03 13:57:49 -05:00
DJP
4190cd48fa Add B2→B1 reset buttons to debug Global campaigns view and fix log message
Fixes:

1. Added B1/B2 Status Buttons to Debug Global Campaigns
   - Shows  Update B1 → B2 button for B1 campaigns
   - Shows 🔄 Reset B2 → B1 button for B2 campaigns
   - Same as debug view for Local campaigns
   - Allows testing status changes

2. Fixed Python Log Message
   - Changed 'Status updated: A1 → A2' to 'B1 → B2'
   - Now correctly reports B1→B2 status update

PHP Test:
1. B1→B2 tab
2. Debug: Load ALL Global Campaigns
3. Find NUTELLA (B1)
4. Click  Update B1 → B2
5. Or click 🔄 Reset B2 → B1 if already B2

Python Test:
Script now logs: 'Status updated: B1 → B2' 

🤖 Generated with Claude Code
2025-11-03 13:56:41 -05:00
DJP
95c3256183 Fix B1→B2 workflow - Correct function name and search for Global comm campaigns
Fixes:

1. PHP: Fixed function name
   - Changed findFinalAssetsFolder() → findUploadFolder()
   - This function already looks for Final Assets folder
   - Now PHP interface works without fatal error

2. Python: Search for Global comm campaigns
   - Added campaign_type parameter to search_campaigns()
   - B1→B2 uses: campaign_type='Global comm'
   - A1→A2 uses: campaign_type='Local Adaptation' (default)

3. Python: Fixed log messages
   - 'Searching for B1 Global campaigns' (not A1)
   - 'No B1 campaigns found' (not A1)

4. Box Folder Configuration
   - B1→B2 uses folder: 349261192115
   - Folder naming: MASTERS_Campaign_Name

B1→B2 Now:
 Searches Global comm campaigns
 Filters for B1 status
 Uses Final Assets folder (05. not 01.)
 Uploads to correct Box folder (349261192115)
 Names folders: MASTERS_NUTELLA_PLANT-BASED_LAUNCH

Test:
1. Refresh PHP app - should load now
2. B1→B2 tab should work
3. Python script should find B1 campaigns

🤖 Generated with Claude Code
2025-11-03 13:47:20 -05:00
DJP
33860decfd Fix B1→B2 workflow - Use Final Assets folder and MASTERS_ Box folder naming
Key Changes:

PHP Interface:
 Added currentTab = 'global-masters' to select_campaign_b1
 Added get_global_master_assets action handler
 Uses findFinalAssetsFolder() (looks for '05. Final Assets')
 Shows selected campaign info
 Displays Global Master assets when found

Python B1→B2 Script:
 Use different Box folder: 349261192115 (not 348304357505)
 Pass is_global=True to get_master_assets()
 Box folder naming: MASTERS_Campaign_Name (no campaign number)
 Folder prefix: MASTERS_ instead of campaign ID

DAM Client:
 Updated get_master_assets() to accept is_global parameter
 If is_global=True: Uses find_final_assets_folder() (05. Final Assets)
 If is_global=False: Uses _find_master_assets_folder() (01. Master Assets)

Configuration:
 Added BOX_ROOT_FOLDER_B1_B2=349261192115
 Three separate Box folders now configured

B1 Workflow Differences:
- Uses '05. Final Assets' folder (not '01. Master Assets')
- Box folder: 349261192115 (not 348304357505)
- Box naming: MASTERS_NUTELLA_PLANT-BASED_LAUNCH
- No campaign number in folder name

Test Next:
1. Refresh PHP app
2. B1→B2 tab → Select NUTELLA campaign
3. Click 'Get Global Master Assets'
4. Should find assets in 05. Final Assets folder

🤖 Generated with Claude Code
2025-11-03 13:39:34 -05:00
DJP
9a32d28389 Create Python b1_to_b2_download.py script for Global Masters workflow
Created complete B1→B2 automation script:
 Based on tested a1_to_a2_download.py structure
 Searches for campaigns with status B1
 Searches Global comm campaigns (not Local Adaptation)
 Downloads Global Master assets from DAM
 Uploads to Box with tracking IDs (folder: 348304357505)
 Stores in PostgreSQL with full metadata
 Updates status B1 → B2 when all successful
 Sends webhook with B1→B2 status change
 Sends email notifications (b1_to_b2_complete, b1_to_b2_partial)
 Log rotation (28 files, 10MB each)
 Single-run mode (process one campaign and exit)

Usage:
cd Python-Version
source venv/bin/activate
python scripts/b1_to_b2_download.py

Cron Setup:
*/5 * * * * python scripts/b1_to_b2_download.py

Test Campaign Available:
- NUTELLA PLANT-BASED LAUNCH
- Folder ID: 676f2bcde4c7bcf7ef783e97f7495069bf50b6bc
- Status: B1

Complete B1→B2 automation ready for testing!

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-03 12:47:48 -05:00