ferrero-opentext/Python-Version/MARKDOWN_DOCS/CUTOVER.md

# Ferrero Automation - Production Cutover Plan

## Overview

This document outlines the complete cutover process from development/staging to production environment for the Ferrero DAM Content Scaling automation system.

**Current State:** Running on staging/dev environment
**Target State:** Full production deployment with live DAM integration

---

## Pre-Cutover Checklist

### 1. Verify Current System Health

**Development Environment:**
```bash
# Check all workflows are working
python scripts/a1_to_a2_download.py
python scripts/a5_to_a6_download.py
python scripts/b1_to_b2_download.py
python scripts/a2_to_a3_upload_polling.py
python scripts/creativex_scoring_storing.py

# Verify database connectivity
python scripts/test_connection.py

# Check recent logs for errors
grep -i error logs/*.log | tail -50
```

**Expected Results:**
- ✅ All scripts execute without errors
- ✅ Database connections successful
- ✅ Box authentication working
- ✅ DAM OAuth2 authentication working
- ✅ Email notifications sending

### 2. Backup Current Configuration

```bash
# Backup current .env file
cp .env .env.backup.$(date +%Y%m%d)

# Backup database
./database/backup.sh --daily

# Backup Box-config.json
cp ../Box-config.json ../Box-config.json.backup.$(date +%Y%m%d)

# Backup current cron jobs
crontab -l > cron_backup_$(date +%Y%m%d).txt
```

### 3. Document Current URLs and IDs

Create reference document with current settings:
```bash
# Extract current settings
grep "^DAM_BASE_URL" .env
grep "^BOX_ROOT_FOLDER" .env
grep "WEBHOOK" .env
```

---

## Production Credentials & Configuration

### Required Production Values

#### 1. DAM Production Endpoints

**Current (Dev/Staging):**
```bash
DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token
```

**Production:**
```bash
DAM_BASE_URL=https://dam.ferrero.com/otmmapi  # VERIFY actual production URL
DAM_AUTH_URL=https://dam.ferrero.com/otdsws/oauth2/token  # VERIFY
```

**ACTION REQUIRED:**
- [ ] Confirm production DAM URLs with Ferrero IT team
- [ ] Obtain production OAuth2 credentials
- [ ] Test authentication against production endpoint

#### 2. DAM OAuth2 Credentials

**Current (Dev):**
```bash
DAM_CLIENT_ID=otds-OLV
DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1
```

**Production:**
```bash
DAM_CLIENT_ID=<production_client_id>
DAM_CLIENT_SECRET=<production_client_secret>
```

**ACTION REQUIRED:**
- [ ] Request production OAuth2 client credentials from Ferrero
- [ ] Test credentials against production DAM
- [ ] Verify token refresh works

#### 3. Box Folder IDs

**Current (Dev/Staging):**
```bash
BOX_ROOT_FOLDER_A1_A2=348304357505      # Local master downloads
BOX_ROOT_FOLDER_A2_A3=348526703108      # Agency upload processing
BOX_ROOT_FOLDER_B1_B2=349261192115      # Global master downloads
BOX_ROOT_FOLDER_CREATIVEX=350605024645  # CreativeX PDF scoring
```

**Production:**
```bash
BOX_ROOT_FOLDER_A1_A2=<production_folder_id>
BOX_ROOT_FOLDER_A2_A3=<production_folder_id>
BOX_ROOT_FOLDER_B1_B2=<production_folder_id>
BOX_ROOT_FOLDER_CREATIVEX=<production_folder_id>
```

**ACTION REQUIRED:**
- [ ] Create production Box folders with correct structure
- [ ] Document folder IDs
- [ ] Set proper permissions for Box service account
- [ ] Test upload/download to each folder

#### 4. Box Authentication

**Current (Dev):**
```bash
BOX_CLIENT_ID=l2atwxxq4xna7phcjr2uifm4mbah69qp
BOX_CLIENT_SECRET=6XcuCQ6akpk9daE0UHaGSv3mSxWaER4l
BOX_JWT_KEY_ID=n1izyn3l
BOX_PASSPHRASE=971585f5fd6171428c14a7c8899af5ab
BOX_ENTERPRISE_ID=43984435
```

**Production:**
```bash
BOX_CLIENT_ID=<production_value>
BOX_CLIENT_SECRET=<production_value>
BOX_JWT_KEY_ID=<production_value>
BOX_PASSPHRASE=<production_value>
BOX_ENTERPRISE_ID=<production_value>
```

**File:** `../Box-config.json` (one level up from Python-Version)

**ACTION REQUIRED:**
- [ ] Request production Box JWT credentials
- [ ] Generate new Box-config.json for production
- [ ] Place at correct location (../Box-config.json)
- [ ] Set permissions: `chmod 600 ../Box-config.json`
- [ ] Test Box authentication

#### 5. Database

**Current (Dev):**
```bash
DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=ferrero_pass_2025
```

**Production Options:**

**Option A: Keep local PostgreSQL (recommended for start)**
```bash
DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=<new_strong_production_password>
```

**Option B: Use managed database service**
```bash
DB_HOST=<rds_or_managed_db_hostname>
DB_PORT=5432
DB_USER=ferrero_prod_user
DB_PASSWORD=<production_password>
```

**ACTION REQUIRED:**
- [ ] Decide: Local PostgreSQL or managed service
- [ ] Change default password (ferrero_pass_2025)
- [ ] Update .env with production credentials
- [ ] Initialize production database with init.sql
- [ ] Set up automated backups (cron jobs)

#### 6. Email Notifications

**Current (Dev):**
```bash
SMTP_SERVER=smtp.mailgun.org
SMTP_USER=twist@mail.dev.oliver.solutions
SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency
ERROR_EMAIL=daveporter@oliver.agency
REPORT_EMAILS=daveporter@oliver.agency
```

**Production:**
```bash
SMTP_SERVER=smtp.mailgun.org
SMTP_USER=<production_mailgun_user>
SENDER_EMAIL=ferrero-automation@ferrero.com  # Or oliver.agency
ERROR_EMAIL=ferrero-it@ferrero.com,daveporter@oliver.agency
REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency
```

**ACTION REQUIRED:**
- [ ] Set up production email domain
- [ ] Update recipient lists for production team
- [ ] Test email delivery
- [ ] Verify spam filters don't block automated emails

#### 7. Webhooks

**Current (Dev):**
```bash
CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/3f9ztwl8qnljufo0l65utfv5wvvnt9m5
```

**Production:**
```bash
CAMPAIGN_STATUS_WEBHOOK_URL=<production_make_scenario_url>
```

**ACTION REQUIRED:**
- [ ] Create production Make.com scenario
- [ ] Update webhook URL
- [ ] Test webhook delivery
- [ ] Verify webhook signature validation (if enabled)

#### 8. CreativeX / LlamaCloud

**Current (Dev):**
```bash
LLAMA_CLOUD_API_KEY=llx-EDmfh0ZReUbXUbaa5i5275TAP2LznNDqc3skJRL3HY4RUDcf
CREATIVEX_AGENT_NAME=Creativex-Extract
```

**Production:**
```bash
LLAMA_CLOUD_API_KEY=<production_api_key>
CREATIVEX_AGENT_NAME=Creativex-Extract  # Same agent or production-specific?
```

**ACTION REQUIRED:**
- [ ] Confirm same LlamaCloud API key for production or get new one
- [ ] Verify agent "Creativex-Extract" exists and works in production
- [ ] Test extraction with production data

#### 9. mTLS Certificate (Optional)

**Current (Dev):**
```bash
DAM_MTLS_BASE_URL=https://dev-auth.app-api.ferrero.com/00003/mm
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx
DAM_MTLS_CERT_PASSWORD=fnJ8xrnh!54NE&2HR62=2P3YEy+hy9RajZ7v5&=y
```

**Production:**
```bash
DAM_MTLS_BASE_URL=https://prod-auth.app-api.ferrero.com/00003/mm  # VERIFY
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx
DAM_MTLS_CERT_PASSWORD=<production_cert_password>
```

**ACTION REQUIRED:**
- [ ] Obtain production mTLS certificate
- [ ] Place in config/certificates/
- [ ] Set permissions: `chmod 600 config/certificates/*.pfx`
- [ ] Whitelist production server IP
- [ ] Test mTLS authentication: `python scripts/test_connection.py --auth-pfx`

---

## Cutover Steps

### Phase 1: Preparation (1-2 days before)

#### Step 1.1: Update Production Server Code

```bash
# SSH to production server
ssh user@production-server

# Navigate to deployment directory
cd /opt/ferrero-automation/Python-Version

# Pull latest code from Bitbucket
git pull origin main

# Verify all new files present
ls -la scripts/creativex_scoring_storing.py
ls -la database/backup.sh
ls -la DATABASE_BACKUP_GUIDE.md
```

#### Step 1.2: Install Dependencies

```bash
# Activate virtual environment
source venv/bin/activate

# Install new dependencies (llama-cloud-services)
sudo venv/bin/pip install llama-cloud-services

# Verify installation
pip show llama-cloud-services

# Or reinstall all dependencies
sudo venv/bin/pip install -r requirements.txt
```

#### Step 1.3: Database Updates

```bash
# Create creativex_scores table
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql

# Or just add the new table
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
CREATE TABLE IF NOT EXISTS creativex_scores (
    id SERIAL PRIMARY KEY,
    filename VARCHAR(500) NOT NULL,
    box_file_id VARCHAR(255),
    creativex_id VARCHAR(255),
    creativex_url TEXT,
    quality_score VARCHAR(50),
    full_extraction_data JSONB,
    extracted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    status VARCHAR(50) DEFAULT 'active',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_creativex_filename ON creativex_scores(filename);
CREATE INDEX IF NOT EXISTS idx_creativex_box_file ON creativex_scores(box_file_id);
CREATE INDEX IF NOT EXISTS idx_creativex_status ON creativex_scores(status);
"

# Verify table created
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores"
```

#### Step 1.4: Update .env with Production Credentials

```bash
# Backup current .env
cp .env .env.staging.backup

# Edit .env with production values
nano .env
```

**Update these sections:**

```bash
# DAM Production URLs
DAM_BASE_URL=<production_dam_url>
DAM_AUTH_URL=<production_auth_url>
DAM_CLIENT_ID=<production_client_id>
DAM_CLIENT_SECRET=<production_client_secret>

# Box Production Folder IDs
BOX_ROOT_FOLDER_A1_A2=<production_folder>
BOX_ROOT_FOLDER_A2_A3=<production_folder>
BOX_ROOT_FOLDER_B1_B2=<production_folder>
BOX_ROOT_FOLDER_CREATIVEX=<production_folder>

# Database Production Password
DB_PASSWORD=<new_strong_password>

# Email Production Recipients
ERROR_EMAIL=ferrero-it@ferrero.com,operations@oliver.agency
REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency

# Webhooks Production URL
CAMPAIGN_STATUS_WEBHOOK_URL=<production_webhook>

# CreativeX Production
LLAMA_CLOUD_API_KEY=<production_key>
```

Save and exit.

#### Step 1.5: Update Box-config.json

```bash
# Backup current Box config
cp ../Box-config.json ../Box-config.json.staging.backup

# Replace with production Box JWT config
nano ../Box-config.json
```

Paste production Box JWT credentials (JSON format from Box Developer Console).

```bash
# Set secure permissions
chmod 600 ../Box-config.json
```

#### Step 1.6: Add mTLS Production Certificate (If Using)

```bash
# Copy production certificate
cp /path/to/dam-mtls-prod.pfx config/certificates/

# Set permissions
chmod 600 config/certificates/dam-mtls-prod.pfx

# Update .env
nano .env
# Update: DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx
# Update: DAM_MTLS_CERT_PASSWORD=<production_password>
```

### Phase 2: Testing (Before Cutover)

#### Step 2.1: Test Database Connection

```bash
# Test PostgreSQL
PGPASSWORD=<new_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"

# Test via Python
python scripts/test_connection.py
```

**Expected:** Connection successful, tables accessible

#### Step 2.2: Test DAM Authentication

```bash
# Test OAuth2 (production credentials)
python scripts/test_connection.py

# Test mTLS (if using)
python scripts/test_connection.py --auth-pfx
```

**Expected:** DAM connection successful, can retrieve campaigns

#### Step 2.3: Test Box Authentication

```bash
# Run Box connection test
python -c "
from shared.config_loader import load_config
from shared.box_client import BoxClient
config = load_config('config/config.yaml')
box = BoxClient(config, root_folder_id=config['box']['root_folder_a1_a2'])
print('Box test:', box.test_connection())
"
```

**Expected:** Box connection OK

#### Step 2.4: Test Email Notifications

```bash
# Send test email
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
notifier.send_email(
    template_name='a1_to_a2_no_assets',
    recipients=config['notifications']['recipients']['success'],
    data={
        'campaign_name': 'TEST CAMPAIGN',
        'campaign_id': 'TEST123',
        'campaign_number': 'C000000999'
    }
)
print('Test email sent')
"
```

**Expected:** Email received by production recipients

#### Step 2.5: Test Webhook

```bash
# Test webhook delivery
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
result = notifier.send_webhook(
    url=config['webhooks']['campaign_status_update']['url'],
    payload={'test': 'cutover_test', 'campaign': 'TEST'}
)
print('Webhook test:', result)
"
```

**Expected:** Webhook received in production Make.com scenario

#### Step 2.6: Test CreativeX Extraction

```bash
# Upload test PDF to production CreativeX folder
# Then run extraction
python scripts/creativex_scoring_storing.py

# Verify in database
PGPASSWORD=<new_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT filename, quality_score FROM creativex_scores WHERE status = 'active' LIMIT 5;
"
```

**Expected:** Scores extracted and stored successfully

#### Step 2.7: Dry Run Workflows (Production Data, No Status Updates)

**Important:** Test with real production campaigns but DON'T update statuses yet

```bash
# Test A1→A2 download (will process but not update status without all assets)
# Manually select a small test campaign first
python scripts/a1_to_a2_download.py

# Check logs
tail -100 logs/a1_to_a2.log

# Verify files in Box production folder
# Verify records in database

# Test A2→A3 upload with test file
# Upload one file with tracking ID to A2→A3 folder
python scripts/a2_to_a3_upload_polling.py

# Verify upload to DAM
# Check email notification
```

**CRITICAL:** During dry run testing:
- ✅ Use real production endpoints
- ✅ Use real production Box folders
- ⚠️ Process SMALL test campaigns only
- ⚠️ Verify you're comfortable with status updates before going live
- ⚠️ Have rollback plan ready

### Phase 3: Go-Live Decision Point

#### Go/No-Go Criteria

✅ **GO if all true:**
- [ ] All connection tests passed
- [ ] Email notifications working
- [ ] Webhooks delivering successfully
- [ ] Test campaign processed without errors
- [ ] Box uploads/downloads working
- [ ] DAM uploads working with correct metadata
- [ ] CreativeX scores extracting and integrating
- [ ] Database backups configured and tested
- [ ] Team trained on monitoring/support
- [ ] Rollback plan documented

⛔ **NO-GO if any true:**
- [ ] Authentication failing
- [ ] Production credentials incomplete
- [ ] Email delivery issues
- [ ] Webhook not reaching Make.com
- [ ] Test uploads corrupted or failed
- [ ] Metadata mapping incorrect
- [ ] No backup system in place

---

## Go-Live Execution

### Phase 4: Cutover Day

#### Step 4.1: Final Preparation (Morning)

```bash
# 1. Stop all cron jobs temporarily
crontab -e
# Comment out all Ferrero automation lines (add # at start)

# 2. Create final backup of staging data
./database/backup.sh --daily

# 3. Verify no workflows are running
ps aux | grep python | grep scripts

# 4. Clear any pending Box files
# Manually verify Box folders are empty or in known state
```

#### Step 4.2: Update Environment Variable

```bash
# Update environment setting
nano .env

# Change:
ENV=staging
# To:
ENV=production

# Save and exit
```

#### Step 4.3: Enable Production Cron Jobs

```bash
crontab -e
```

**Production Cron Schedule:**
```cron
# A1→A2: Download master assets from DAM (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron_a1_a2.log 2>&1

# A5→A6: Download rejected assets for rework (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a5_to_a6_download.py >> logs/cron_a5_a6.log 2>&1

# B1→B2: Download global master campaigns (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/b1_to_b2_download.py >> logs/cron_b1_b2.log 2>&1

# A2→A3: Process agency uploads from Box (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a2_to_a3_upload_polling.py >> logs/cron_a2_a3.log 2>&1

# Daily Report: Send summary email (7:00 PM daily)
0 19 * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/daily_report.py >> logs/daily_report.log 2>&1

# Database Backup: Daily at 2:00 AM
0 2 * * * cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --daily >> logs/backup.log 2>&1

# Database Backup: Weekly on Sundays at 3:00 AM
0 3 * * 0 cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --weekly >> logs/backup.log 2>&1

# Backup Health Check: Daily at 8:00 AM
0 8 * * * cd /opt/ferrero-automation/Python-Version && ./database/check_backups.sh --quiet >> logs/backup_check.log 2>&1
```

Save and exit.

#### Step 4.4: Monitor Initial Runs

```bash
# Watch logs in real-time
tail -f logs/cron_a1_a2.log

# In another terminal, watch all workflow logs
tail -f logs/*.log

# Monitor for 1-2 hours
# Look for:
# - ✅ "No A1 campaigns found" (normal if no campaigns ready)
# - ✅ Successful processing messages
# - ⚠️ Any errors or authentication failures
```

#### Step 4.5: Verify First Campaign Processing

**When first A1 campaign appears:**

```bash
# Monitor A1→A2 processing
tail -f logs/a1_to_a2.log

# Expected flow:
# - Campaign detected
# - Assets downloaded from DAM
# - Uploaded to Box with tracking IDs
# - Stored in database
# - Status updated A1→A2
# - Email sent

# Verify in database
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT tracking_id, original_filename, created_at FROM master_assets ORDER BY created_at DESC LIMIT 10;
"

# Verify in Box folder
# Check production Box folder contains files with tracking IDs
```

---

## Post-Cutover Monitoring

### First 24 Hours

#### Hourly Checks:

```bash
# Check cron execution
tail -100 logs/cron_*.log

# Check for errors
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"

# Check database growth
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    (SELECT COUNT(*) FROM master_assets) as masters,
    (SELECT COUNT(*) FROM derivative_assets) as derivatives,
    (SELECT COUNT(*) FROM creativex_scores) as creativex,
    (SELECT COUNT(*) FROM campaign_status) as campaigns;
"

# Check Box folders
# Verify files are being created/deleted as expected
```

#### Email Monitoring:

- ✅ Success emails arriving
- ⚠️ No error emails (or expected errors only)
- 📊 Daily report at 7 PM contains production data

#### Workflow Health:

```bash
# Check all workflows completed recent runs
ls -lt logs/*.log | head -10

# Verify campaigns progressing through stages
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT campaign_number, status, webhook_sent, updated_at
FROM campaign_status
ORDER BY updated_at DESC
LIMIT 10;
"
```

### First Week

#### Daily Checks:

1. **Review daily report email** (7 PM)
   - Verify statistics are accurate
   - Check success rates
   - Review any errors

2. **Check backup health**
   ```bash
   ./database/check_backups.sh
   ```

3. **Monitor disk space**
   ```bash
   df -h /opt/ferrero-automation
   du -sh Python-Version/backups/
   du -sh Python-Version/logs/
   du -sh Python-Version/temp/
   ```

4. **Check cron logs**
   ```bash
   # Verify cron is executing
   grep CRON /var/log/syslog | grep ferrero | tail -20
   ```

#### Weekly Tasks:

- [ ] Review workflow success rates
- [ ] Check for any recurring errors
- [ ] Verify webhook delivery to Make.com
- [ ] Test restore from backup (monthly, but good to test early)
- [ ] Review with Ferrero team

---

## Rollback Plan

### If Critical Issues Found

#### Quick Rollback (Revert to Staging)

```bash
# 1. Stop production cron jobs
crontab -e
# Comment out all lines

# 2. Restore staging .env
cp .env.staging.backup .env

# 3. Restore staging Box-config.json
cp ../Box-config.json.staging.backup ../Box-config.json

# 4. Update environment
nano .env
# Change: ENV=production → ENV=staging

# 5. Restart staging cron jobs
crontab -e
# Restore original cron jobs
```

#### Database Rollback

```bash
# If database needs rollback
./database/restore.sh backups/dumps/<pre_cutover_backup>.sql.gz
```

#### Code Rollback

```bash
# If code changes need rollback
git log --oneline | head -10  # Find commit before cutover
git revert <commit_hash>
git push origin main
```

---

## Known Production Differences

### Development vs Production

| Component | Development | Production | Notes |
|-----------|-------------|------------|-------|
| **DAM URL** | ppr.dam.ferrero.com | dam.ferrero.com (TBD) | Confirm with Ferrero IT |
| **Box Folders** | Dev folder IDs | Prod folder IDs | Create new folders |
| **OAuth2 Creds** | Dev credentials | Prod credentials | Request from Ferrero |
| **Email Recipients** | daveporter@oliver.agency | Ferrero team + ops | Update recipient lists |
| **Webhook URL** | Dev Make scenario | Prod Make scenario | Create new scenario |
| **DB Password** | ferrero_pass_2025 | Strong production password | Change for security |
| **mTLS Cert** | dev-mtls-dev.pfx | dam-mtls-prod.pfx | Get prod certificate |
| **Server IP** | Dev IP | Prod IP | Whitelist for mTLS |

---

## Cutover Communication Plan

### Stakeholders to Notify

**Before Cutover (48 hours):**
- [ ] Ferrero DAM team
- [ ] Ferrero IT team
- [ ] Oliver Agency operations
- [ ] Content team users

**Email Template:**
```
Subject: Ferrero DAM Automation - Production Cutover [Date]

The automated content scaling system will be going live on [DATE] at [TIME].

What's changing:
- Automation moves from staging to production DAM environment
- Campaigns will automatically progress through workflow stages
- Email notifications will come from production system

What to expect:
- A1→A2: Master assets auto-downloaded to Box
- A2→A3: Localized assets auto-uploaded to DAM
- A5→A6: Rejected assets flagged for rework
- CreativeX scores automatically attached to assets

Support during cutover:
- [Contact person]: [Email/Phone]
- Monitoring: First 24 hours continuous
- Rollback available if issues found

Please report any issues immediately to [ERROR_EMAIL]
```

**During Cutover:**
- [ ] Status updates every 2 hours
- [ ] Available for immediate response

**After Cutover (24 hours):**
- [ ] Summary email with statistics
- [ ] Known issues (if any)
- [ ] Next steps

---

## Post-Cutover Validation

### Validation Checklist (First 24 Hours)

#### Workflow Validations:

**A1→A2 (Master Download):**
- [ ] Campaign detected with status A1
- [ ] All assets downloaded from DAM
- [ ] Uploaded to Box with tracking IDs
- [ ] Stored in database with full metadata
- [ ] Status updated to A2
- [ ] Success email sent
- [ ] Tracking IDs are unique
- [ ] Subfolder structure preserved

**A5→A6 (Rejections/Rework):**
- [ ] Only "NOT APPROVED" assets downloaded
- [ ] Rejection comments extracted
- [ ] Tracking IDs reused correctly
- [ ] Email lists rejection reasons
- [ ] Approved assets skipped

**B1→B2 (Global Masters):**
- [ ] Only "Global comm" campaigns processed
- [ ] Uploaded to separate Box folder
- [ ] MASTERS_ prefix in folder names
- [ ] Email sent (no webhook)

**A2→A3 (Agency Uploads):**
- [ ] Files detected in Box A2→A3 folder
- [ ] Tracking IDs parsed from filenames
- [ ] Master metadata loaded from database
- [ ] CreativeX scores looked up from database
- [ ] Default values used when scores missing
- [ ] Uploaded to correct DAM folder
- [ ] Files deleted from Box after success
- [ ] Email shows CreativeX status

**CreativeX Scoring:**
- [ ] PDFs processed from Box folder 350605024645
- [ ] Scores extracted via LlamaExtract
- [ ] Stored in database with version tracking
- [ ] Decimals removed from IDs and scores
- [ ] PDFs deleted after successful extraction
- [ ] Email sent with version badges

#### Database Validations:

```bash
# Check all tables have data
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    'master_assets' as table_name, COUNT(*) as records FROM master_assets
UNION ALL SELECT
    'derivative_assets', COUNT(*) FROM derivative_assets
UNION ALL SELECT
    'creativex_scores', COUNT(*) FROM creativex_scores
UNION ALL SELECT
    'campaign_status', COUNT(*) FROM campaign_status
UNION ALL SELECT
    'asset_events', COUNT(*) FROM asset_events;
"

# Check for any errors in logs
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"
```

#### Backup Validations:

```bash
# Verify first production backup
./database/check_backups.sh

# Expected:
# - Backup created at 2:00 AM
# - Backup file size reasonable
# - No errors in backup.log
```

---

## Production Support Procedures

### Monitoring Dashboard Commands

**Create monitoring script:** `scripts/production_status.sh`

```bash
#!/bin/bash
echo "========================================="
echo "Ferrero Automation Production Status"
echo "========================================="
echo ""

# Database stats
echo "Database Records:"
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    (SELECT COUNT(*) FROM master_assets WHERE status = 'active') as active_masters,
    (SELECT COUNT(*) FROM creativex_scores WHERE status = 'active') as active_scores,
    (SELECT COUNT(*) FROM campaign_status WHERE webhook_sent = true) as completed_campaigns;
"

echo ""
echo "Recent Activity (Last 24 hours):"
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT COUNT(*) as new_assets_today
FROM master_assets
WHERE created_at > NOW() - INTERVAL '24 hours';
"

echo ""
echo "Backup Status:"
./database/check_backups.sh --quiet

echo ""
echo "Recent Errors:"
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)" | tail -5
```

### Emergency Contacts

**Escalation Path:**
1. **Level 1:** Operations team (monitors daily)
2. **Level 2:** Technical lead (Dave Porter)
3. **Level 3:** Ferrero IT team

**Contact Information:**
- Operations: operations@oliver.agency
- Technical Lead: daveporter@oliver.agency
- Ferrero IT: [TO BE PROVIDED]

---

## Configuration Checklist Summary

### Files to Update:

- [ ] `Python-Version/.env` - All production credentials
- [ ] `../Box-config.json` - Production Box JWT
- [ ] `config/certificates/dam-mtls-prod.pfx` - Production mTLS cert (if using)
- [ ] Crontab - Production schedule
- [ ] `/etc/hosts` - Production hostname if needed

### Credentials to Obtain:

- [ ] Production DAM OAuth2 client ID and secret
- [ ] Production DAM base URL and auth URL
- [ ] Production Box folder IDs (4 folders)
- [ ] Production Box JWT credentials
- [ ] Production mTLS certificate (if using)
- [ ] Production webhook URL (Make.com)
- [ ] Production email recipients list
- [ ] Production LlamaCloud API key (or confirm dev key works)

### Testing to Complete:

- [ ] DAM connection test
- [ ] Box connection test
- [ ] Database connection test
- [ ] Email delivery test
- [ ] Webhook delivery test
- [ ] CreativeX extraction test
- [ ] End-to-end workflow test with small campaign
- [ ] Backup and restore test

---

## Success Metrics

### Week 1 Targets:

- **Uptime:** > 99% (workflows running without crashes)
- **Success Rate:** > 95% (assets processed successfully)
- **Email Delivery:** 100% (all notifications received)
- **Backup Success:** 7/7 daily backups completed
- **Response Time:** < 10 minutes (cron cycle)

### Monitor These KPIs:

```bash
# Check success rate from daily report emails
# Or query database:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    COUNT(*) FILTER (WHERE webhook_sent = true) as successful_campaigns,
    COUNT(*) as total_campaigns,
    ROUND(100.0 * COUNT(*) FILTER (WHERE webhook_sent = true) / NULLIF(COUNT(*), 0), 1) as success_rate
FROM campaign_status
WHERE created_at > NOW() - INTERVAL '7 days';
"
```

---

## Troubleshooting Production Issues

### Issue: Workflows Not Running

```bash
# Check cron service
systemctl status cron

# Check cron logs
grep CRON /var/log/syslog | tail -50

# Manually run workflow
cd /opt/ferrero-automation/Python-Version
source venv/bin/activate
python scripts/a1_to_a2_download.py
```

### Issue: Authentication Failing

```bash
# Test OAuth2
python scripts/test_connection.py

# Test mTLS
python scripts/test_connection.py --auth-pfx

# Check credentials in .env
grep "DAM_" .env | grep -v PASSWORD
```

### Issue: Database Not Accessible

```bash
# Check PostgreSQL container
docker ps | grep ferrero-tracking-db

# Check logs
docker logs ferrero-tracking-db | tail -50

# Test connection
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT 1;"
```

### Issue: Box Upload/Download Failing

```bash
# Test Box connection
python -c "
from shared.config_loader import load_config
from shared.box_client import BoxClient
config = load_config('config/config.yaml')
box = BoxClient(config)
print(box.test_connection())
"

# Check Box-config.json location
ls -la ../Box-config.json

# Verify Box folder IDs
grep "BOX_ROOT_FOLDER" .env
```

### Issue: Emails Not Sending

```bash
# Check SMTP settings
grep "SMTP" .env

# Test email manually
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
# Send test email...
"

# Check Mailgun logs (if using Mailgun)
```

---

## Appendix: Production Readiness Checklist

### Infrastructure:

- [ ] Production server provisioned and accessible
- [ ] Docker installed and running
- [ ] PostgreSQL container running on port 5437
- [ ] Sufficient disk space (minimum 10 GB)
- [ ] Network access to DAM, Box, email servers
- [ ] Firewall rules configured (if using mTLS)
- [ ] SSL certificates valid (if applicable)

### Code Deployment:

- [ ] Latest code pulled from Bitbucket
- [ ] Virtual environment created (Python 3.10+)
- [ ] All dependencies installed (requirements.txt)
- [ ] llama-cloud-services installed
- [ ] Scripts executable (chmod +x)
- [ ] Database schema initialized (init.sql)
- [ ] creativex_scores table created

### Configuration:

- [ ] .env file configured with production values
- [ ] Box-config.json in correct location (../Box-config.json)
- [ ] mTLS certificate in place (if using)
- [ ] All folder IDs updated
- [ ] Email recipients updated
- [ ] Webhook URL updated

### Testing:

- [ ] All connection tests passed
- [ ] Test campaign processed successfully
- [ ] Email notifications received
- [ ] Webhook delivery confirmed
- [ ] CreativeX extraction working
- [ ] Backup system tested

### Automation:

- [ ] Cron jobs configured
- [ ] Cron service running
- [ ] Log rotation configured
- [ ] Backup schedule active
- [ ] Monitoring scripts in place

### Documentation:

- [ ] Team trained on system
- [ ] Support procedures documented
- [ ] Rollback plan understood
- [ ] Escalation path defined
- [ ] Emergency contacts listed

### Sign-off:

- [ ] Technical lead approval: _________________ Date: _______
- [ ] Operations approval: _________________ Date: _______
- [ ] Ferrero stakeholder approval: _________________ Date: _______

---

## Post-Cutover: Continuous Improvement

### Month 1 Review:

- Analyze success rates
- Identify recurring errors
- Optimize cron schedule if needed
- Tune email notifications
- Adjust backup retention if needed

### Potential Enhancements:

- [ ] Add Grafana/Prometheus monitoring
- [ ] Implement alerting thresholds
- [ ] Add Slack notifications
- [ ] Remote backup to S3/cloud storage
- [ ] Point-in-time recovery (WAL archiving)
- [ ] High availability setup (if needed)
- [ ] Load balancing (if volume increases)

---

**Document Version:** 1.0
**Last Updated:** November 11, 2025
**Owner:** Dave Porter (daveporter@oliver.agency)
**Review Date:** [After cutover + 1 week]