ferrero-opentext/Python-Version/README.md

# Ferrero Content Scaling - Python Automation System

**Complete automated workflow for Ferrero DAM Content Scaling**

**Version:** 2.1
**Last Updated:** April 16, 2026
**Status:** ✅ Production Ready & Fully Tested

---

## 📋 Table of Contents

1. [Overview](#overview)
2. [What's Included](#whats-included)
3. [Quick Start](#quick-start)
4. [Server Deployment](#server-deployment)
5. [Workflows](#workflows)
6. [Authentication](#authentication)
7. [Configuration](#configuration)
8. [Monitoring](#monitoring)
9. [Troubleshooting](#troubleshooting)

---

## 🎯 Overview

**Automated end-to-end workflow system that:**
- Downloads master assets from OpenText DAM
- Uploads to Box with tracking IDs
- Processes localized assets from Box
- Filters rejected assets for rework
- Sends email notifications
- Generates daily summary reports
- Supports OAuth2 and mTLS authentication

**All workflows run autonomously via cron (every 5 minutes)**

---

## 📦 What's Included

### 4 Workflow Automation Scripts

1. **A1→A2** - Master Assets Download (Local Adaptation)
   - Downloads master assets from DAM
   - Uploads to Box with tracking IDs (never start with 'M')
   - Stores in database with full metadata
   - Updates status A1 → A2

2. **A5→A6** - Rework Assets Download (Rejections)
   - **Filters for NOT APPROVED assets only**
   - Extracts rejection comments from 3 reviewers
   - Uploads to separate Revisions Box folder
   - Smart tracking ID reuse
   - Updates status A5 → A6

3. **B1→B2** - Global Master Assets Download
   - Downloads global master assets
   - Uses MASTERS_CampaignNumber naming
   - Separate Box folder for global assets
   - Tracking IDs always start with 'M'
   - Updates status B1 → B2

4. **A2→A3** - Upload from Box (Polling)
   - Monitors Box folder for agency uploads
   - Matches tracking IDs to master assets
   - Uploads to DAM with inherited metadata
   - Updates status A2 → A3

5. **CreativeX Scoring** - Independent Service
   - Runs as a separate service (every 15 mins)
   - Extracts scores from PDFs using LlamaCloud
   - Stores scores in database
   - Independent of main orchestrator to prevent blocking

### Additional Scripts

5. **Daily Report** - Automated summary email (7 PM daily)
   - Analyzes all workflow logs (last 24 hours)
   - Campaign and asset statistics
   - Success rate calculations
   - Error summaries
   - Professional dashboard-style email

6. **Test Scripts**
   - `test_connection.py` - Tests DAM, Box, Database connections
   - `test_mtls_cert.py` - Tests mTLS certificate loading
   - `test_mtls_debug.py` - Detailed mTLS debugging

---

## 🚀 Quick Start

### Prerequisites

- Python 3.6+ (server) or 3.10+ (local)
- PostgreSQL 15+ database
- Box account with JWT app
- OpenText DAM API credentials
- SMTP server for emails (Mailgun recommended)

### Local Setup

```bash
cd Python-Version

# 1. Create virtual environment
python3 -m venv venv
source venv/bin/activate

# 2. Install dependencies
pip install -r requirements.txt

# 3. Configure environment
cp .env.example .env  # If exists, or edit .env directly
nano .env

# 4. Test connections
python scripts/test_connection.py

# Should show:
# ✓ DAM connection OK
# ✓ Box connection OK
# ✓ Database connection OK
```

---

## 🖥️ Server Deployment

### Step 1: Copy Files to Server

**Important: Box-config.json location**

The Box JWT configuration file must be **one folder up** from Python-Version:

```
/opt/ferrero-automation/
├── Box-config.json          ← Box JWT config HERE
└── Python-Version/
    ├── .env
    ├── config/
    │   └── config.yaml      ← Points to ../Box-config.json
    └── scripts/
```

**Copy command:**
```bash
# Copy both folders
scp Box-config.json user@server:/opt/ferrero-automation/
scp -r Python-Version/ user@server:/opt/ferrero-automation/
```

**Or use rsync:**
```bash
# From Ferrero-Opentext folder
rsync -av Box-config.json Python-Version/ user@server:/opt/ferrero-automation/
```

### Step 2: Setup Database

**Using Docker (Recommended):**
```bash
cd /opt/ferrero-automation/Python-Version

docker run -d \
  --name ferrero-tracking-db \
  --restart always \
  -e POSTGRES_DB=ferrero_tracking \
  -e POSTGRES_USER=ferrero_user \
  -e POSTGRES_PASSWORD=ferrero_pass_2025 \
  -p 5437:5432 \
  -v $(pwd)/database/init.sql:/docker-entrypoint-initdb.d/init.sql \
  -v ferrero-db-data:/var/lib/postgresql/data \
  postgres:15

# Wait 10 seconds for initialization
sleep 10

# Verify database
docker exec ferrero-tracking-db psql -U ferrero_user -d ferrero_tracking -c "\dt"
```

### Step 3: Install Python Dependencies

```bash
cd /opt/ferrero-automation/Python-Version

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

### Step 4: Configure for Server

**Update .env:**
```bash
nano .env

# Update database (if different from defaults)
DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=ferrero_pass_2025  # CHANGE IN PRODUCTION!

# Verify all other credentials are correct
```

### Step 5: Test Everything

```bash
source venv/bin/activate

# Test connections
python scripts/test_connection.py

# Test each workflow (won't process if no campaigns)
python scripts/a1_to_a2_download.py
python scripts/a5_to_a6_download.py
python scripts/b1_to_b2_download.py

# Generate test report
python scripts/daily_report.py
```

### Step 6: Setup Cron Jobs

```bash
crontab -e
```

**Add:**
```cron
# Ferrero Automation (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron_a1_a2.log 2>&1
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a5_to_a6_download.py >> logs/cron_a5_a6.log 2>&1
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/b1_to_b2_download.py >> logs/cron_b1_b2.log 2>&1
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a2_to_a3_upload_polling.py >> logs/cron_a2_a3.log 2>&1

# Daily report (7 PM)
0 19 * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/daily_report.py >> logs/daily_report.log 2>&1
```

---

## 🔄 Workflows

### A1→A2: Master Assets Download

**Purpose:** Download master assets from DAM for local adaptation

**Process:**
1. Search for campaigns with status = A1
2. Process first campaign only (cron runs every 5 min)
3. Recursively search Master Assets folder (including subfolders)
4. Download all assets from DAM
5. Upload to Box folder `348304357505`
   - Folder name: `C000000078-Campaign_Name`
   - Preserves DAM subfolder structure
6. Store in database with tracking ID (never starts with 'M' for regular files)
7. **ONLY update A1→A2 if ALL assets successful**
8. Send webhook + email notification

**Box Folder:** 348304357505 (Local Adaptation)
**Email:** a1_to_a2_complete, a1_to_a2_partial, a1_to_a2_no_assets

---

### A5→A6: Rework Assets Download (NOT APPROVED Filter)

**Purpose:** Download ONLY rejected assets that need rework

**Process:**
1. Search for campaigns with status = A5
2. Process first campaign only
3. Recursively search Final Assets folder
4. **Filter for ECOMMERCE STATUS = "NOT APPROVED" only**
5. Skip all approved/other status assets
6. Download NOT APPROVED assets
7. Extract rejection details:
   - Approver comments, certifier, date
   - Legal comments, certifier, date
   - IA&CC comments, certifier, date
8. Upload to Box folder `349441822875` (Revisions)
   - Folder name: `C000000078-Campaign_Name-Revisions`
   - Preserves DAM subfolder structure
9. Smart tracking ID lookup (reuse if asset exists from A1→A2)
10. **ONLY update A5→A6 if ALL rejected assets successful**
11. Send detailed email with rejection comments

**Box Folder:** 349441822875 (Revisions)
**Email:** a5_to_a6_rejections, a5_to_a6_partial, a5_to_a6_no_rejections

**Special Features:**
- ✅ Filters assets by ECOMMERCE STATUS field
- ✅ Extracts 10 rejection-related fields
- ✅ Shows detailed comments in email (color-coded by reviewer)
- ✅ Reuses tracking IDs from original download

---

### B1→B2: Global Master Assets Download

**Purpose:** Download global master assets for global campaigns

**Process:**
1. Search for campaigns with status = B1 AND type = "Global comm"
2. Process first campaign only
3. Recursively search Final Assets folder (B1 uses Final, not Master)
4. Download all assets
5. Upload to Box folder `349261192115`
   - Folder name: `MASTERS_C000000068-Campaign_Name`
   - Preserves DAM subfolder structure
6. Store in database with tracking ID (always starts with 'M' for master files)
7. **ONLY update B1→B2 if ALL assets successful**
8. Send email notification (no webhook)

**Box Folder:** 349261192115 (Global Masters)
**Email:** b1_to_b2_complete, b1_to_b2_partial, b1_to_b2_no_assets

---

### A2→A3: Upload from Box

**Purpose:** Process localized assets uploaded by agency to Box

**Process:**
1. Poll Box folder `348526703108` for new files
2. Match tracking ID from filename
3. Load master metadata from database
4. Build asset representation (27 MVP fields)
5. Update Description, Language, State from filename
6. Upload to DAM Final Assets folder
7. Delete file from Box
8. **Update A2→A3 when ALL campaign assets uploaded**

**Two tracking modes:**
- **Full inheritance** (standard): Inherits all metadata from the master asset
- **Folder-only** (`-N` suffix): Builds metadata from a reference template (`config/asset_representation_template.json`) and populates values from the filename. Used when the derivative only needs the upload folder from the master.

**Asset type overrides:** Certain asset types (e.g., EOL) trigger field overrides configured in the environment's field mappings file (e.g., Agency Name, Languages, IP Rights, validity dates).

**Box Folder:** 348526703108 (Agency Uploads)
**Email:** a2_to_a3_file_uploaded, a2_to_a3_complete

---

### Daily Report

**Purpose:** Automated daily summary email at 7 PM

**Process:**
1. Analyzes all 4 workflow logs (last 24 hours)
2. Calculates statistics:
   - Campaigns found/processed/completed/partial
   - Assets successful/failed/skipped
   - NOT APPROVED count (A5→A6)
   - Success rate percentage
   - Error count
3. Per-workflow breakdown
4. Campaign details (collapsible)
5. Error logs (collapsible)
6. Professional dashboard-style email

**Runs:** Daily at 7:00 PM via cron
**Email:** Blue dashboard theme with statistics

---

## 🔐 Authentication

Both OAuth2 and mTLS authentication are supported across **all workflows**.

### OAuth2 (Default - Production Ready)

**Current production authentication method** - No special configuration needed.

**Configuration (.env):**
```bash
DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token
DAM_CLIENT_ID=otds-OLV
DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1
```

**Usage (default - no flag needed):**
```bash
# All workflows use OAuth2 by default
python scripts/a1_to_a2_download.py
python scripts/a2_to_a3_upload_polling.py
python scripts/a4_webhook_monitor.py
python scripts/a5_to_a6_download.py
python scripts/b1_to_b2_download.py
```

**How it works:**
- Requests OAuth2 token from `/otdsws/oauth2/token`
- Uses client_credentials grant type
- Token added to `Authorization: Bearer` header
- Token auto-refreshes (1 hour expiry)
- No IP whitelisting required

---

### mTLS Certificate (Optional - Enhanced Security)

**Client certificate authentication** - Requires IP whitelisting.

**Configuration (.env):**
```bash
DAM_MTLS_BASE_URL=https://dev-auth.app-api.ferrero.com/00003/mm
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx
DAM_MTLS_CERT_PASSWORD=fnJ8xrnh!54NE&2HR62=2P3YEy+hy9RajZ7v5&=y
```

**Usage (add --auth-pfx flag):**
```bash
# All workflows support mTLS with --auth-pfx flag
python scripts/a1_to_a2_download.py --auth-pfx
python scripts/a2_to_a3_upload_polling.py --auth-pfx
python scripts/a4_webhook_monitor.py --auth-pfx
python scripts/a5_to_a6_download.py --auth-pfx
python scripts/b1_to_b2_download.py --auth-pfx
```

**How it works:**
- Uses PFX/P12 client certificate
- Certificate auto-converted to PEM format
- Presented during TLS handshake
- **Different base URL** than OAuth2
- **Requires IP whitelisting** on Ferrero's side

**Requirements:**
- ⚠️ **Must run from whitelisted IP address**
- ⚠️ **Different API endpoint:** `https://dev-auth.app-api.ferrero.com/00003/mm`
- ⚠️ **For production:** Get production certificate (current is dev)

**Testing mTLS:**
```bash
# Test certificate loading
python scripts/test_mtls_cert.py

# Should show:
# ✓ Certificate loaded successfully
# ✓ Private key extracted
# ✓ Certificate chain: 3 certificates
# ✓ Valid PEM format

# Test DAM connection with mTLS
python scripts/test_connection.py --auth-pfx

# Should show:
# ✓ DAM connection OK (mTLS)
# ✓ Box connection OK
# ✓ Database connection OK
```

### Authentication Comparison

| Feature | OAuth2 (Default) | mTLS (--auth-pfx) |
|---------|------------------|-------------------|
| **Base URL** | `https://ppr.dam.ferrero.com/otmmapi` | `https://dev-auth.app-api.ferrero.com/00003/mm` |
| **IP Whitelisting** | Not required | **Required** |
| **Certificate** | Not needed | PFX/P12 file required |
| **Production Ready** | ✅ Yes | ⚠️ Need production cert |
| **Flag** | None (default) | `--auth-pfx` |
| **Token Refresh** | Automatic | Not applicable |

### Auth V2 (Hybrid mTLS/OAuth) - Recommended for Large Files

**New hybrid authentication method** - Uses mTLS to get OAuth token, then direct DAM access.
**Solves APIM file size limits (500MB+)**

**Configuration (.env):**
```bash
DAM_MTLS_OAUTH_URL=https://dev-auth.app-api.ferrero.com/00003/mm/v6_OAUTH
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx
DAM_MTLS_CERT_PASSWORD=your_password
```

**Usage (add --auth-pfx-v2 flag):**
```bash
python scripts/a1_to_a2_download.py --auth-pfx-v2
```

**How it works:**
1. Authenticates with mTLS cert to `DAM_MTLS_OAUTH_URL`
2. Receives OAuth bearer token
3. Connects DIRECTLY to DAM (bypassing APIM gateway)
4. Uploads/Downloads large files without size limits

---

## 🎼 Orchestration (New)

**Recommended way to run all workflows.**

Instead of managing multiple cron jobs, use the **Orchestrator**:
- Runs every minute
- Manages schedules for all scripts
- **Prevents overlaps** (locks execution)
- Handles Daily Report schedule (7 PM)
- Logs to `logs/orchestrator.log`

**Usage:**
```bash
# Run manually (force all tasks)
python scripts/orchestrator.py --force

# Run via Cron (Standard)
python scripts/orchestrator.py
```

**Configuration:**
Modify `scripts/orchestrator.py` `TASKS` list to change intervals.

---

## 🖥️ Server Deployment (Updated)

### Step 1-5: Same as above

### Step 6: Setup Cron Job (Orchestrator)

**You only need ONE cron entry now:**

```bash
crontab -e
```

**Add:**
```cron
# Ferrero Orchestrator (Runs every minute)
* * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/orchestrator.py >> logs/orchestrator_cron.log 2>&1
```

### Option 2: Systemd Services (Recommended for Production)

We now have dedicated systemd services for all components:

1. **Orchestrator Service** (Main workflows)
   ```bash
   sudo cp ferrero-orchestrator-prod.service /etc/systemd/system/
   sudo systemctl enable --now ferrero-orchestrator-prod
   ```

2. **CreativeX Service** (Scoring)
   ```bash
   sudo cp scripts/creativex-service.service /etc/systemd/system/
   sudo systemctl enable --now creativex-service
   ```

3. **Database Backups** (Daily & Weekly)
   ```bash
   sudo cp ferrero-backup-*.{service,timer} /etc/systemd/system/
   sudo systemctl enable --now ferrero-backup-daily.timer
   sudo systemctl enable --now ferrero-backup-weekly.timer
   ```

**View Logs:**
```bash
# Orchestrator
tail -f logs/orchestrator_prod.log

# CreativeX
tail -f logs/creativex_service.log

# Backups
sudo journalctl -u ferrero-backup-daily.service
```

---

## 🎛️ Script Options & Flags

### A1→A2 Download Script

```bash
# OAuth2 (default)
python scripts/a1_to_a2_download.py

# mTLS certificate authentication
python scripts/a1_to_a2_download.py --auth-pfx

# Help
python scripts/a1_to_a2_download.py --help
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication instead of OAuth2

---

### A2→A3 Upload Script

```bash
# OAuth2 (default)
python scripts/a2_to_a3_upload_polling.py

# mTLS authentication
python scripts/a2_to_a3_upload_polling.py --auth-pfx

# Keep files in Box after upload (testing)
python scripts/a2_to_a3_upload_polling.py --keep-files

# Force status update A2→A3 (testing)
python scripts/a2_to_a3_upload_polling.py --A3update

# Combination (mTLS + keep files + force update)
python scripts/a2_to_a3_upload_polling.py --auth-pfx --keep-files --A3update

# Help
python scripts/a2_to_a3_upload_polling.py --help
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication
- `--keep-files` - Don't delete files from Box after upload (for testing)
- `--A3update` - Force campaign status update A2→A3 after upload (for testing)

---

### A4 Webhook Monitor Script

```bash
# OAuth2 (default)
python scripts/a4_webhook_monitor.py

# mTLS authentication
python scripts/a4_webhook_monitor.py --auth-pfx

# Help
python scripts/a4_webhook_monitor.py --help
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication

**Behavior:**
- Processes ALL A4 campaigns in one run
- Prevents duplicate webhooks via database tracking
- Sends "Live Campaign: NO" webhook
- Shows summary statistics

---

### A5→A6 Download Script

```bash
# OAuth2 (default)
python scripts/a5_to_a6_download.py

# mTLS authentication
python scripts/a5_to_a6_download.py --auth-pfx

# Help
python scripts/a5_to_a6_download.py --help
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication

**Special Features:**
- Filters for `ECOMMERCE STATUS = "NOT APPROVED"` only
- Extracts rejection comments from 3 reviewers
- Reuses tracking IDs from A1→A2 if asset exists

---

### B1→B2 Download Script

```bash
# OAuth2 (default)
python scripts/b1_to_b2_download.py

# mTLS authentication
python scripts/b1_to_b2_download.py --auth-pfx

# Help
python scripts/b1_to_b2_download.py --help
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication

**Special Features:**
- Filters for campaigns with type = "Global comm"
- Uses "MASTERS_" prefix for Box folders
- Searches Final Assets (not Master Assets)

---

### Daily Report Script

```bash
# Generate daily report
python scripts/daily_report.py

# No authentication flags - analyzes local logs only
```

**No flags needed** - This script only reads local log files and doesn't connect to DAM.

---

### Test Connection Script

```bash
# Test all connections with OAuth2
python scripts/test_connection.py

# Test all connections with mTLS
python scripts/test_connection.py --auth-pfx
```

**Available Flags:**
- `--auth-pfx` - Test mTLS certificate authentication

**Tests:**
- ✅ DAM connection (OAuth2 or mTLS)
- ✅ Box connection
- ✅ Database connection

---

### Reset Campaign to A1 (Interactive)

**Purpose:** Reset any campaign from any A# status back to A1 for testing/retrying workflows.

```bash
# Reset any A# campaigns to A1 (shows all A2, A3, A4, A5, A6)
python scripts/reset_campaign_to_a1.py

# Reset only A4 campaigns to A1
python scripts/reset_campaign_to_a1.py --status A4

# With mTLS authentication
python scripts/reset_campaign_to_a1.py --auth-pfx

# Filter A2 campaigns with mTLS
python scripts/reset_campaign_to_a1.py --status A2 --auth-pfx
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication
- `--status A2` - Filter for specific status (A2, A3, A4, A5, A6)

**Interactive Behavior:**
- Shows campaign name, number, and current status
- Asks yes/no for each campaign individually
- Waits for user input before proceeding to next
- Shows summary at the end

**Example Output:**
```
======================================================================
Campaign 1/2
======================================================================
Campaign: CONTENT SCALING 12345
Number:   C000000078
Current:  A4
======================================================================
Reset this campaign to A1? (yes/no): yes

Resetting campaign to A1...
✓ SUCCESS: Campaign reset to A1
```

---

### Advance A1 to A3 (Interactive)

**Purpose:** Advance A1 campaigns directly to A3, skipping A2. Useful for testing A2→A3 upload workflow.

```bash
# Advance A1 campaigns to A3
python scripts/advance_a1_to_a3.py

# With mTLS authentication
python scripts/advance_a1_to_a3.py --auth-pfx
```

**Available Flags:**
- `--auth-pfx` - Use mTLS certificate authentication

**Interactive Behavior:**
- Finds all A1 campaigns
- Shows campaign name and number
- Asks yes/no for each campaign individually
- Advances to A3 if user confirms
- Shows summary at the end

**Example Output:**
```
======================================================================
Campaign 1/3
======================================================================
Campaign: CONTENT SCALING TEST
Number:   C000000551
Current:  A1
Target:   A3
======================================================================
Advance this campaign A1 → A3? (yes/no): yes

Advancing campaign A1 → A3...
✓ SUCCESS: Campaign advanced to A3
```

**Use Case:**
- Testing A2→A3 upload workflow without running full A1→A2 download
- Quickly setting up test campaigns in A3 status
- Skipping the download step when you already have files in Box

---

### Update Campaign Status (Targeted)

**Purpose:** Update a specific campaign to a specific status. Search by campaign number or name.

```bash
# Update specific campaign by number
python scripts/update_campaign_status.py --camp C000000078 --status A2

# Update by partial campaign name
python scripts/update_campaign_status.py --camp "CONTENT SCALING" --status A4

# Update to A3 with mTLS
python scripts/update_campaign_status.py --camp C000000551 --status A3 --auth-pfx

# Reset to A1
python scripts/update_campaign_status.py --camp "KINDER JOY" --status A1
```

**Required Flags:**
- `--camp` - Campaign number (e.g., C000000078) or partial campaign name
- `--status` - Target status (A1, A2, A3, A4, A5, A6, B1, B2)

**Optional Flags:**
- `--auth-pfx` - Use mTLS certificate authentication

**Interactive Behavior:**
- Searches all statuses for matching campaign
- Shows current status and target status
- If multiple matches, lets user choose which one
- Asks for yes/no confirmation before updating
- Shows success/failure result

**Example Output:**
```
======================================================================
Found 1 matching campaign(s)
======================================================================

======================================================================
Selected Campaign
======================================================================
Name:           CONTENT SCALING KINDER JOY
Number:         C000000553
Current Status: A4
Target Status:  A1
======================================================================

Update campaign status A4 → A1? (yes/no): yes

Updating campaign status...

======================================================================
✓ SUCCESS
======================================================================
Campaign:    CONTENT SCALING KINDER JOY
Number:      C000000553
New Status:  A1
======================================================================
```

**Use Cases:**
- Quickly update specific campaign without searching through lists
- Testing workflow progression: A1 → A2 → A3 → A4
- Resetting specific campaigns for retesting
- Works with campaign number OR campaign name (partial match)

---

## ⚙️ Configuration

### Environment Variables (.env)

**Required variables:**

```bash
# Environment
ENV=staging  # or production

# DAM - OAuth2 (default)
DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token
DAM_CLIENT_ID=otds-OLV
DAM_CLIENT_SECRET=your_secret_here

# DAM - mTLS (optional with --auth-pfx)
DAM_MTLS_BASE_URL=https://dev-auth.app-api.ferrero.com/00003/mm
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx
DAM_MTLS_CERT_PASSWORD=your_cert_password

# Box
BOX_CLIENT_ID=your_box_client_id
BOX_CLIENT_SECRET=your_box_secret
BOX_JWT_KEY_ID=your_jwt_key
BOX_PASSPHRASE=your_passphrase
BOX_ENTERPRISE_ID=43984435

# Box Folders
BOX_ROOT_FOLDER_A1_A2=348304357505  # Local Adaptation downloads
BOX_ROOT_FOLDER_A2_A3=348526703108  # Agency uploads
BOX_ROOT_FOLDER_B1_B2=349261192115  # Global Masters

# Database
DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=ferrero_pass_2025

# Email (Mailgun SMTP)
SMTP_SERVER=smtp.mailgun.org
SMTP_PORT=587
SMTP_USER=your_smtp_user
SMTP_PASSWORD=your_smtp_password
SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency
ERROR_EMAIL=daveporter@oliver.agency
REPORT_EMAILS=daveporter@oliver.agency

# Webhook
CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/your_webhook_id
```

### Box JWT Configuration

**File location:** `../Box-config.json` (one folder up from Python-Version)

**Structure:**
```
Ferrero-Opentext/
├── Box-config.json          ← Box JWT config
└── Python-Version/
    └── config/
        └── config.yaml      ← References ../Box-config.json
```

**For server deployment**, copy both:
```bash
scp Box-config.json user@server:/opt/ferrero-automation/
scp -r Python-Version/ user@server:/opt/ferrero-automation/
```

### Field Mappings (Environment-Specific)

The system auto-detects the environment from the DAM URL and loads the appropriate config:

- **PPR:** `config/field_mappings_ppr.yaml` (pre-production, `ppr.dam.ferrero.com`)
- **PROD:** `config/field_mappings_prod.yaml` (production, `dam.ferrero.com`)

Each file defines: MVP fields, filename update rules, forced values, defaults, and asset type overrides.

### Asset Type Mappings

`config/asset_type_mappings.yaml` maps 3-letter codes from the naming tool to DAM domain values (e.g., `EHI` -> `heroimage`, `EOL` -> `externallegalopinion`).

**Last updated:** April 16, 2026 per Scaling Agencies Metadata List. 38 asset types mapped (was 39). Changes:
- **Removed:** CID, ECB, EBS, EOP, EUG, EWB, FPO, PKI, PRI
- **Added:** EAN, ESI, NTB, PIR, PKC, PKT, SCP, SNC, UPI
- **Changed:** DAT DAM code updated from `digitalassettoolkit` to `digitalasset`

### Asset Representation Template

`config/asset_representation_template.json` is the reference template for folder-only mode (`-N` flag uploads). It contains the full field metadata structure that the DAM API requires for asset creation. This template was provided by the client and should be updated if the DAM metadata model changes.

### Asset Type Overrides (EOL / LTD)

Certain asset types trigger field overrides configured in the field mappings file. Currently configured for both PPR and PROD:

**EOL (External Legal Opinion)**
- Agency Name = "-"
- Production House = "-"
- Main Languages = "Global"
- IP Rights = "Yes"
- Licensing = "No"
- Validity dates removed

**LTD (Licensing Translation Document)** — supports the EOL workflow with translated license claims. Same overrides as EOL, plus a fixed Description: `"Translation of License claim - For approval purposes only"`. Currently mapped to the same DAM-side code (`externallegalopinion`) as a placeholder pending client confirmation.

These overrides are applied after all other field processing and take final precedence. An empty-string override removes the field; a non-empty override targeting a field that isn't in `mvp_fields` will be appended as a simple string field.

---

## 🗄️ Database

### Connection Details

**Default (Docker):**
- Host: localhost
- Port: 5437
- Database: ferrero_tracking
- User: ferrero_user
- Password: ferrero_pass_2025 (⚠️ CHANGE IN PRODUCTION!)

### Schema

**Tables:**
1. **master_assets** (35 columns)
   - tracking_id, opentext_id, full_metadata (JSONB)
   - global_master_campaign_id, local_campaign_id
   - All metadata fields
2. **derivative_assets** - Localized assets
3. **asset_events** - Audit log
4. **workflow_state** - Workflow tracking

**Complete schema:** See `DATABASE_SCHEMA.md` or `database/init.sql`

### Setup Database

```bash
# Docker method
docker run -d \
  --name ferrero-tracking-db \
  --restart always \
  -e POSTGRES_DB=ferrero_tracking \
  -e POSTGRES_USER=ferrero_user \
  -e POSTGRES_PASSWORD=ferrero_pass_2025 \
  -p 5437:5432 \
  -v $(pwd)/database/init.sql:/docker-entrypoint-initdb.d/init.sql \
  -v ferrero-db-data:/var/lib/postgresql/data \
  postgres:15

# Or native PostgreSQL
psql -h localhost -U postgres -f database/init.sql
```

---

## 📊 Monitoring

### Log Files

All logs in `logs/` directory:

- `a1_to_a2.log` - A1→A2 workflow
- `a5_to_a6.log` - A5→A6 workflow (rework)
- `b1_to_b2.log` - B1→B2 workflow (global)
- `a2_to_a3.log` - A2→A3 workflow (upload)
- `cron_*.log` - Cron execution logs
- `daily_report.log` - Report generation

**Log rotation:**
- Max size: 10MB per file
- Backups: 28 files (~1 month retention)
- Auto-cleanup

**View logs:**
```bash
tail -f logs/a1_to_a2.log
tail -f logs/a5_to_a6.log

# Search for errors
grep -i error logs/*.log
grep -i "NOT APPROVED" logs/a5_to_a6.log
```

### Database Queries

**Connect to database:**
```bash
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking
```

**Quick queries (from command line):**

```bash
# View recent A1→A2 downloads
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename, brand_code, country_code, local_campaign_id, created_at FROM master_assets ORDER BY created_at DESC LIMIT 10;"

# View full details for recent assets
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT id, tracking_id, original_filename, file_extension, file_size_bytes, brand_code, country_code, language_code, local_campaign_id, global_master_campaign_id, created_at FROM master_assets ORDER BY created_at DESC LIMIT 20;"

# Count total assets
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) as total_assets FROM master_assets WHERE status = 'active';"

# Assets by campaign
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT local_campaign_id, COUNT(*) as asset_count FROM master_assets WHERE status = 'active' GROUP BY local_campaign_id ORDER BY asset_count DESC;"

# View campaign status (webhooks)
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT campaign_number, campaign_name, status, live_campaign, webhook_sent, webhook_sent_at FROM campaign_status ORDER BY created_at DESC LIMIT 10;"

# View derivative assets (A2→A3 uploads)
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, derivative_filename, upload_status, uploaded_at FROM derivative_assets ORDER BY created_at DESC LIMIT 10;"

# View recent audit events
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, event_type, event_timestamp FROM asset_events ORDER BY event_timestamp DESC LIMIT 20;"
```

**Advanced queries (in psql):**

```sql
-- Assets with full metadata
SELECT tracking_id, original_filename,
       full_metadata->>'name' as metadata_name,
       jsonb_array_length(full_metadata->'metadata'->'metadata_element_list') as field_count,
       created_at
FROM master_assets
ORDER BY created_at DESC
LIMIT 10;

-- Global master relationships
SELECT local_campaign_id, global_master_campaign_id, COUNT(*) as asset_count
FROM master_assets
WHERE global_master_campaign_id IS NOT NULL
GROUP BY local_campaign_id, global_master_campaign_id
ORDER BY asset_count DESC;

-- Tracking ID with master and derivatives
SELECT
    m.tracking_id,
    m.original_filename as master_filename,
    m.local_campaign_id,
    COUNT(d.id) as derivative_count,
    m.created_at
FROM master_assets m
LEFT JOIN derivative_assets d ON m.tracking_id = d.tracking_id
GROUP BY m.tracking_id, m.original_filename, m.local_campaign_id, m.created_at
ORDER BY m.created_at DESC
LIMIT 20;

-- Assets by brand
SELECT brand_code, COUNT(*) as count
FROM master_assets
WHERE status = 'active'
GROUP BY brand_code
ORDER BY count DESC;

-- Check for duplicates
SELECT opentext_id, local_campaign_id, COUNT(*)
FROM master_assets
GROUP BY opentext_id, local_campaign_id
HAVING COUNT(*) > 1;

-- Recent workflow activity
SELECT workflow_name, campaign_id, last_status, last_run_at
FROM workflow_state
ORDER BY last_run_at DESC
LIMIT 10;
```

**Database management:**

```bash
# Initialize/reset database
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql

# Backup database
pg_dump -h localhost -p 5437 -U ferrero_user ferrero_tracking > backup_$(date +%Y%m%d).sql

# View table sizes
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\dt+"

# View all tables
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\dt"
```

### Email Notifications

**All workflows send emails:**
- ✅ Success (green theme)
- ⚠️ Partial/failures (orange theme)
- ❌ Errors (red theme)
- 📊 Daily report (blue dashboard)

**Templates include:**
- Campaign details
- Asset lists with tracking IDs
- Box URLs
- Rejection comments (A5→A6)
- Error messages
- Next steps

---

## 🔧 Troubleshooting

### Connection Test Fails

```bash
# Run diagnostic
python scripts/test_connection.py

# Check each component
python scripts/test_connection.py --auth-pfx  # Test mTLS
```

**Common issues:**
- ❌ DAM: Check credentials in .env
- ❌ Box: Verify Box-config.json is one folder up
- ❌ Database: Check PostgreSQL is running on port 5437
- ❌ mTLS: Check you're on whitelisted IP

### Box JWT File Not Found

**Error:** `Failed to initialize Box client: [Errno 2] No such file or directory: '../Box-config.json'`

**Solution:**
```bash
# Check Box-config.json location
ls -la ../Box-config.json

# Should be one folder up from Python-Version
# Structure:
#   /opt/ferrero-automation/
#   ├── Box-config.json
#   └── Python-Version/
```

### Database Connection Fails

```bash
# Check Docker container running
docker ps | grep ferrero

# Check PostgreSQL service
sudo systemctl status postgresql

# Test connection manually
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT 1;"
```

### No Campaigns Found

**Expected behavior** - scripts check every 5 minutes:
```
No A1 campaigns found - exiting
```

This is normal! The script will check again in 5 minutes.

### Email Not Sending

```bash
# Check SMTP configuration in .env
grep SMTP .env

# Test email manually
python -c "from scripts.shared.notifier import Notifier; from shared.config_loader import load_config; n=Notifier(load_config('config/config.yaml')); print('OK')"
```

### mTLS Certificate Issues

```bash
# Test certificate loads
python scripts/test_mtls_cert.py

# Should show:
# ✓ Certificate loaded successfully
# ✓ Valid PEM format

# Check permissions
ls -l config/certificates/*.pfx
# Should be: -rw------- (600)

# Fix if needed
chmod 600 config/certificates/*.pfx
```

### Logs Not Created

```bash
# Create log directories
mkdir -p logs/backup
chmod 755 logs
```

---

## 📁 Complete File Structure

```
/opt/ferrero-automation/
├── Box-config.json                    ← Box JWT (one folder up!)
└── Python-Version/
    ├── .env                          ← Environment variables
    ├── config/
    │   ├── config.yaml              ← Main configuration
    │   ├── field_mappings.yaml      ← Field definitions
    │   └── certificates/            ← mTLS certificates (gitignored)
    │       └── dam-mtls-dev.pfx
    ├── database/
    │   └── init.sql                 ← Database schema
    ├── scripts/
    │   ├── a1_to_a2_download.py    ← A1→A2 workflow
    │   ├── a5_to_a6_download.py    ← A5→A6 workflow (rework)
    │   ├── b1_to_b2_download.py    ← B1→B2 workflow (global)
    │   ├── a2_to_a3_upload_polling.py  ← A2→A3 workflow
    │   ├── daily_report.py          ← Daily summary
    │   ├── test_connection.py       ← Connection tester
    │   ├── test_mtls_cert.py        ← Certificate tester
    │   ├── test_mtls_debug.py       ← mTLS debugger
    │   └── shared/
    │       ├── dam_client.py        ← DAM API (dual auth)
    │       ├── box_client.py        ← Box API
    │       ├── database.py          ← PostgreSQL
    │       ├── notifier.py          ← Email + webhooks (12 templates)
    │       └── config_loader.py     ← Config loader
    ├── logs/                        ← Auto-created
    ├── temp/                        ← Auto-created
    ├── venv/                        ← Virtual environment
    ├── requirements.txt             ← Python dependencies
    ├── DEPLOYMENT_GUIDE.md          ← Full deployment guide
    ├── DATABASE_SCHEMA.md           ← Schema documentation
    └── README.md                    ← This file
```

---

## 🔒 Security

### Production Checklist

**Before going live:**

- [ ] Change database password from default
- [ ] Update all .env passwords
- [ ] Get production mTLS certificate (if using mTLS)
- [ ] Verify Box JWT credentials are production keys
- [ ] Set file permissions:
  ```bash
  chmod 600 .env
  chmod 600 config/certificates/*.pfx
  chmod 600 ../Box-config.json
  ```
- [ ] Configure firewall rules
- [ ] Enable database backups
- [ ] Set up log monitoring
- [ ] Test all workflows end-to-end

### Gitignored Files (Never Committed)

```
.env                     # All credentials
config/certificates/     # mTLS certificates
*.pfx, *.p12            # Certificate files
logs/                    # Log files
temp/                    # Temporary downloads
__pycache__/            # Python cache
*.pyc                    # Compiled files
```

---

## 🎯 Key Features

### Recursive Folder Search (All Workflows)
- ✅ Searches subfolders within Master/Final Assets folders
- ✅ Preserves folder structure in Box
- ✅ Logs subfolder paths

### NOT APPROVED Filtering (A5→A6 Only)
- ✅ Only downloads assets with `ECOMMERCE STATUS = "NOT APPROVED"`
- ✅ Skips approved/other status assets
- ✅ Extracts detailed rejection comments from 3 reviewers

### Smart Tracking ID Management
- ✅ 6-character unique IDs
- ✅ Reuses existing IDs for rework assets (A5→A6)
- ✅ Links master assets to derivatives
- ✅ Full audit trail in database

### Professional Email Notifications
- ✅ Unified modern styling across all templates
- ✅ Color-coded by severity (green/orange/red/blue)
- ✅ Detailed asset lists with Box URLs
- ✅ Rejection comments with reviewer names (A5→A6)
- ✅ Collapsible sections in daily report

---

## 📞 Support

### Getting Help

1. **Check logs:** `tail -f logs/a1_to_a2.log`
2. **Test connections:** `python scripts/test_connection.py`
3. **Check database:** `psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking`
4. **Review config:** `cat config/config.yaml`

### Common Commands

```bash
# Activate environment
source venv/bin/activate

# Test everything
python scripts/test_connection.py

# Manual workflow run
python scripts/a1_to_a2_download.py

# Generate report
python scripts/daily_report.py

# Check cron logs
tail -f logs/cron_*.log

# Database stats
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"
```

---

## 📝 Additional Documentation

- **DEPLOYMENT_GUIDE.md** - Complete server deployment guide
- **DATABASE_SCHEMA.md** - Full database schema reference
- **config/field_mappings.yaml** - Editable field definitions

---

## ✅ Production Ready

**Fully tested and working:**
- ✅ All 4 workflows automated
- ✅ Recursive folder search
- ✅ NOT APPROVED filtering
- ✅ Email notifications (12 templates)
- ✅ Daily summary reports
- ✅ Dual authentication (OAuth2 + mTLS)
- ✅ Complete database schema
- ✅ Deployment documentation

**Compatible with:**
- Python 3.6+ (server)
- Python 3.10+ (local development)
- PostgreSQL 15+
- Box SDK 3.x
- OpenText DAM API v6

---

**Version:** 2.1 - Production Ready
**Last Updated:** April 16, 2026
**Repository:** bitbucket.org:zlalani/ferrero-opentext.git

🚀 **Ready to deploy!**