diff --git a/Python-Version/CUTOVER.md b/Python-Version/CUTOVER.md new file mode 100644 index 0000000..5551f71 --- /dev/null +++ b/Python-Version/CUTOVER.md @@ -0,0 +1,1253 @@ +# Ferrero Automation - Production Cutover Plan + +## Overview + +This document outlines the complete cutover process from development/staging to production environment for the Ferrero DAM Content Scaling automation system. + +**Current State:** Running on staging/dev environment +**Target State:** Full production deployment with live DAM integration + +--- + +## Pre-Cutover Checklist + +### 1. Verify Current System Health + +**Development Environment:** +```bash +# Check all workflows are working +python scripts/a1_to_a2_download.py +python scripts/a5_to_a6_download.py +python scripts/b1_to_b2_download.py +python scripts/a2_to_a3_upload_polling.py +python scripts/creativex_scoring_storing.py + +# Verify database connectivity +python scripts/test_connection.py + +# Check recent logs for errors +grep -i error logs/*.log | tail -50 +``` + +**Expected Results:** +- ✅ All scripts execute without errors +- ✅ Database connections successful +- ✅ Box authentication working +- ✅ DAM OAuth2 authentication working +- ✅ Email notifications sending + +### 2. Backup Current Configuration + +```bash +# Backup current .env file +cp .env .env.backup.$(date +%Y%m%d) + +# Backup database +./database/backup.sh --daily + +# Backup Box-config.json +cp ../Box-config.json ../Box-config.json.backup.$(date +%Y%m%d) + +# Backup current cron jobs +crontab -l > cron_backup_$(date +%Y%m%d).txt +``` + +### 3. Document Current URLs and IDs + +Create reference document with current settings: +```bash +# Extract current settings +grep "^DAM_BASE_URL" .env +grep "^BOX_ROOT_FOLDER" .env +grep "WEBHOOK" .env +``` + +--- + +## Production Credentials & Configuration + +### Required Production Values + +#### 1. DAM Production Endpoints + +**Current (Dev/Staging):** +```bash +DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi +DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token +``` + +**Production:** +```bash +DAM_BASE_URL=https://dam.ferrero.com/otmmapi # VERIFY actual production URL +DAM_AUTH_URL=https://dam.ferrero.com/otdsws/oauth2/token # VERIFY +``` + +**ACTION REQUIRED:** +- [ ] Confirm production DAM URLs with Ferrero IT team +- [ ] Obtain production OAuth2 credentials +- [ ] Test authentication against production endpoint + +#### 2. DAM OAuth2 Credentials + +**Current (Dev):** +```bash +DAM_CLIENT_ID=otds-OLV +DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1 +``` + +**Production:** +```bash +DAM_CLIENT_ID= +DAM_CLIENT_SECRET= +``` + +**ACTION REQUIRED:** +- [ ] Request production OAuth2 client credentials from Ferrero +- [ ] Test credentials against production DAM +- [ ] Verify token refresh works + +#### 3. Box Folder IDs + +**Current (Dev/Staging):** +```bash +BOX_ROOT_FOLDER_A1_A2=348304357505 # Local master downloads +BOX_ROOT_FOLDER_A2_A3=348526703108 # Agency upload processing +BOX_ROOT_FOLDER_B1_B2=349261192115 # Global master downloads +BOX_ROOT_FOLDER_CREATIVEX=350605024645 # CreativeX PDF scoring +``` + +**Production:** +```bash +BOX_ROOT_FOLDER_A1_A2= +BOX_ROOT_FOLDER_A2_A3= +BOX_ROOT_FOLDER_B1_B2= +BOX_ROOT_FOLDER_CREATIVEX= +``` + +**ACTION REQUIRED:** +- [ ] Create production Box folders with correct structure +- [ ] Document folder IDs +- [ ] Set proper permissions for Box service account +- [ ] Test upload/download to each folder + +#### 4. Box Authentication + +**Current (Dev):** +```bash +BOX_CLIENT_ID=l2atwxxq4xna7phcjr2uifm4mbah69qp +BOX_CLIENT_SECRET=6XcuCQ6akpk9daE0UHaGSv3mSxWaER4l +BOX_JWT_KEY_ID=n1izyn3l +BOX_PASSPHRASE=971585f5fd6171428c14a7c8899af5ab +BOX_ENTERPRISE_ID=43984435 +``` + +**Production:** +```bash +BOX_CLIENT_ID= +BOX_CLIENT_SECRET= +BOX_JWT_KEY_ID= +BOX_PASSPHRASE= +BOX_ENTERPRISE_ID= +``` + +**File:** `../Box-config.json` (one level up from Python-Version) + +**ACTION REQUIRED:** +- [ ] Request production Box JWT credentials +- [ ] Generate new Box-config.json for production +- [ ] Place at correct location (../Box-config.json) +- [ ] Set permissions: `chmod 600 ../Box-config.json` +- [ ] Test Box authentication + +#### 5. Database + +**Current (Dev):** +```bash +DB_HOST=localhost +DB_PORT=5437 +DB_USER=ferrero_user +DB_PASSWORD=ferrero_pass_2025 +``` + +**Production Options:** + +**Option A: Keep local PostgreSQL (recommended for start)** +```bash +DB_HOST=localhost +DB_PORT=5437 +DB_USER=ferrero_user +DB_PASSWORD= +``` + +**Option B: Use managed database service** +```bash +DB_HOST= +DB_PORT=5432 +DB_USER=ferrero_prod_user +DB_PASSWORD= +``` + +**ACTION REQUIRED:** +- [ ] Decide: Local PostgreSQL or managed service +- [ ] Change default password (ferrero_pass_2025) +- [ ] Update .env with production credentials +- [ ] Initialize production database with init.sql +- [ ] Set up automated backups (cron jobs) + +#### 6. Email Notifications + +**Current (Dev):** +```bash +SMTP_SERVER=smtp.mailgun.org +SMTP_USER=twist@mail.dev.oliver.solutions +SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency +ERROR_EMAIL=daveporter@oliver.agency +REPORT_EMAILS=daveporter@oliver.agency +``` + +**Production:** +```bash +SMTP_SERVER=smtp.mailgun.org +SMTP_USER= +SENDER_EMAIL=ferrero-automation@ferrero.com # Or oliver.agency +ERROR_EMAIL=ferrero-it@ferrero.com,daveporter@oliver.agency +REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency +``` + +**ACTION REQUIRED:** +- [ ] Set up production email domain +- [ ] Update recipient lists for production team +- [ ] Test email delivery +- [ ] Verify spam filters don't block automated emails + +#### 7. Webhooks + +**Current (Dev):** +```bash +CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/3f9ztwl8qnljufo0l65utfv5wvvnt9m5 +``` + +**Production:** +```bash +CAMPAIGN_STATUS_WEBHOOK_URL= +``` + +**ACTION REQUIRED:** +- [ ] Create production Make.com scenario +- [ ] Update webhook URL +- [ ] Test webhook delivery +- [ ] Verify webhook signature validation (if enabled) + +#### 8. CreativeX / LlamaCloud + +**Current (Dev):** +```bash +LLAMA_CLOUD_API_KEY=llx-EDmfh0ZReUbXUbaa5i5275TAP2LznNDqc3skJRL3HY4RUDcf +CREATIVEX_AGENT_NAME=Creativex-Extract +``` + +**Production:** +```bash +LLAMA_CLOUD_API_KEY= +CREATIVEX_AGENT_NAME=Creativex-Extract # Same agent or production-specific? +``` + +**ACTION REQUIRED:** +- [ ] Confirm same LlamaCloud API key for production or get new one +- [ ] Verify agent "Creativex-Extract" exists and works in production +- [ ] Test extraction with production data + +#### 9. mTLS Certificate (Optional) + +**Current (Dev):** +```bash +DAM_MTLS_BASE_URL=https://dev-auth.app-api.ferrero.com/00003/mm +DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx +DAM_MTLS_CERT_PASSWORD=fnJ8xrnh!54NE&2HR62=2P3YEy+hy9RajZ7v5&=y +``` + +**Production:** +```bash +DAM_MTLS_BASE_URL=https://prod-auth.app-api.ferrero.com/00003/mm # VERIFY +DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx +DAM_MTLS_CERT_PASSWORD= +``` + +**ACTION REQUIRED:** +- [ ] Obtain production mTLS certificate +- [ ] Place in config/certificates/ +- [ ] Set permissions: `chmod 600 config/certificates/*.pfx` +- [ ] Whitelist production server IP +- [ ] Test mTLS authentication: `python scripts/test_connection.py --auth-pfx` + +--- + +## Cutover Steps + +### Phase 1: Preparation (1-2 days before) + +#### Step 1.1: Update Production Server Code + +```bash +# SSH to production server +ssh user@production-server + +# Navigate to deployment directory +cd /opt/ferrero-automation/Python-Version + +# Pull latest code from Bitbucket +git pull origin main + +# Verify all new files present +ls -la scripts/creativex_scoring_storing.py +ls -la database/backup.sh +ls -la DATABASE_BACKUP_GUIDE.md +``` + +#### Step 1.2: Install Dependencies + +```bash +# Activate virtual environment +source venv/bin/activate + +# Install new dependencies (llama-cloud-services) +sudo venv/bin/pip install llama-cloud-services + +# Verify installation +pip show llama-cloud-services + +# Or reinstall all dependencies +sudo venv/bin/pip install -r requirements.txt +``` + +#### Step 1.3: Database Updates + +```bash +# Create creativex_scores table +PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql + +# Or just add the new table +PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +CREATE TABLE IF NOT EXISTS creativex_scores ( + id SERIAL PRIMARY KEY, + filename VARCHAR(500) NOT NULL, + box_file_id VARCHAR(255), + creativex_id VARCHAR(255), + creativex_url TEXT, + quality_score VARCHAR(50), + full_extraction_data JSONB, + extracted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + status VARCHAR(50) DEFAULT 'active', + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_creativex_filename ON creativex_scores(filename); +CREATE INDEX IF NOT EXISTS idx_creativex_box_file ON creativex_scores(box_file_id); +CREATE INDEX IF NOT EXISTS idx_creativex_status ON creativex_scores(status); +" + +# Verify table created +PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores" +``` + +#### Step 1.4: Update .env with Production Credentials + +```bash +# Backup current .env +cp .env .env.staging.backup + +# Edit .env with production values +nano .env +``` + +**Update these sections:** + +```bash +# DAM Production URLs +DAM_BASE_URL= +DAM_AUTH_URL= +DAM_CLIENT_ID= +DAM_CLIENT_SECRET= + +# Box Production Folder IDs +BOX_ROOT_FOLDER_A1_A2= +BOX_ROOT_FOLDER_A2_A3= +BOX_ROOT_FOLDER_B1_B2= +BOX_ROOT_FOLDER_CREATIVEX= + +# Database Production Password +DB_PASSWORD= + +# Email Production Recipients +ERROR_EMAIL=ferrero-it@ferrero.com,operations@oliver.agency +REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency + +# Webhooks Production URL +CAMPAIGN_STATUS_WEBHOOK_URL= + +# CreativeX Production +LLAMA_CLOUD_API_KEY= +``` + +Save and exit. + +#### Step 1.5: Update Box-config.json + +```bash +# Backup current Box config +cp ../Box-config.json ../Box-config.json.staging.backup + +# Replace with production Box JWT config +nano ../Box-config.json +``` + +Paste production Box JWT credentials (JSON format from Box Developer Console). + +```bash +# Set secure permissions +chmod 600 ../Box-config.json +``` + +#### Step 1.6: Add mTLS Production Certificate (If Using) + +```bash +# Copy production certificate +cp /path/to/dam-mtls-prod.pfx config/certificates/ + +# Set permissions +chmod 600 config/certificates/dam-mtls-prod.pfx + +# Update .env +nano .env +# Update: DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx +# Update: DAM_MTLS_CERT_PASSWORD= +``` + +### Phase 2: Testing (Before Cutover) + +#### Step 2.1: Test Database Connection + +```bash +# Test PostgreSQL +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;" + +# Test via Python +python scripts/test_connection.py +``` + +**Expected:** Connection successful, tables accessible + +#### Step 2.2: Test DAM Authentication + +```bash +# Test OAuth2 (production credentials) +python scripts/test_connection.py + +# Test mTLS (if using) +python scripts/test_connection.py --auth-pfx +``` + +**Expected:** DAM connection successful, can retrieve campaigns + +#### Step 2.3: Test Box Authentication + +```bash +# Run Box connection test +python -c " +from shared.config_loader import load_config +from shared.box_client import BoxClient +config = load_config('config/config.yaml') +box = BoxClient(config, root_folder_id=config['box']['root_folder_a1_a2']) +print('Box test:', box.test_connection()) +" +``` + +**Expected:** Box connection OK + +#### Step 2.4: Test Email Notifications + +```bash +# Send test email +python -c " +from shared.config_loader import load_config +from shared.notifier import Notifier +config = load_config('config/config.yaml') +notifier = Notifier(config) +notifier.send_email( + template_name='a1_to_a2_no_assets', + recipients=config['notifications']['recipients']['success'], + data={ + 'campaign_name': 'TEST CAMPAIGN', + 'campaign_id': 'TEST123', + 'campaign_number': 'C000000999' + } +) +print('Test email sent') +" +``` + +**Expected:** Email received by production recipients + +#### Step 2.5: Test Webhook + +```bash +# Test webhook delivery +python -c " +from shared.config_loader import load_config +from shared.notifier import Notifier +config = load_config('config/config.yaml') +notifier = Notifier(config) +result = notifier.send_webhook( + url=config['webhooks']['campaign_status_update']['url'], + payload={'test': 'cutover_test', 'campaign': 'TEST'} +) +print('Webhook test:', result) +" +``` + +**Expected:** Webhook received in production Make.com scenario + +#### Step 2.6: Test CreativeX Extraction + +```bash +# Upload test PDF to production CreativeX folder +# Then run extraction +python scripts/creativex_scoring_storing.py + +# Verify in database +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT filename, quality_score FROM creativex_scores WHERE status = 'active' LIMIT 5; +" +``` + +**Expected:** Scores extracted and stored successfully + +#### Step 2.7: Dry Run Workflows (Production Data, No Status Updates) + +**Important:** Test with real production campaigns but DON'T update statuses yet + +```bash +# Test A1→A2 download (will process but not update status without all assets) +# Manually select a small test campaign first +python scripts/a1_to_a2_download.py + +# Check logs +tail -100 logs/a1_to_a2.log + +# Verify files in Box production folder +# Verify records in database + +# Test A2→A3 upload with test file +# Upload one file with tracking ID to A2→A3 folder +python scripts/a2_to_a3_upload_polling.py + +# Verify upload to DAM +# Check email notification +``` + +**CRITICAL:** During dry run testing: +- ✅ Use real production endpoints +- ✅ Use real production Box folders +- ⚠️ Process SMALL test campaigns only +- ⚠️ Verify you're comfortable with status updates before going live +- ⚠️ Have rollback plan ready + +### Phase 3: Go-Live Decision Point + +#### Go/No-Go Criteria + +✅ **GO if all true:** +- [ ] All connection tests passed +- [ ] Email notifications working +- [ ] Webhooks delivering successfully +- [ ] Test campaign processed without errors +- [ ] Box uploads/downloads working +- [ ] DAM uploads working with correct metadata +- [ ] CreativeX scores extracting and integrating +- [ ] Database backups configured and tested +- [ ] Team trained on monitoring/support +- [ ] Rollback plan documented + +⛔ **NO-GO if any true:** +- [ ] Authentication failing +- [ ] Production credentials incomplete +- [ ] Email delivery issues +- [ ] Webhook not reaching Make.com +- [ ] Test uploads corrupted or failed +- [ ] Metadata mapping incorrect +- [ ] No backup system in place + +--- + +## Go-Live Execution + +### Phase 4: Cutover Day + +#### Step 4.1: Final Preparation (Morning) + +```bash +# 1. Stop all cron jobs temporarily +crontab -e +# Comment out all Ferrero automation lines (add # at start) + +# 2. Create final backup of staging data +./database/backup.sh --daily + +# 3. Verify no workflows are running +ps aux | grep python | grep scripts + +# 4. Clear any pending Box files +# Manually verify Box folders are empty or in known state +``` + +#### Step 4.2: Update Environment Variable + +```bash +# Update environment setting +nano .env + +# Change: +ENV=staging +# To: +ENV=production + +# Save and exit +``` + +#### Step 4.3: Enable Production Cron Jobs + +```bash +crontab -e +``` + +**Production Cron Schedule:** +```cron +# A1→A2: Download master assets from DAM (every 5 minutes) +*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron_a1_a2.log 2>&1 + +# A5→A6: Download rejected assets for rework (every 5 minutes) +*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a5_to_a6_download.py >> logs/cron_a5_a6.log 2>&1 + +# B1→B2: Download global master campaigns (every 5 minutes) +*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/b1_to_b2_download.py >> logs/cron_b1_b2.log 2>&1 + +# A2→A3: Process agency uploads from Box (every 5 minutes) +*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a2_to_a3_upload_polling.py >> logs/cron_a2_a3.log 2>&1 + +# Daily Report: Send summary email (7:00 PM daily) +0 19 * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/daily_report.py >> logs/daily_report.log 2>&1 + +# Database Backup: Daily at 2:00 AM +0 2 * * * cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --daily >> logs/backup.log 2>&1 + +# Database Backup: Weekly on Sundays at 3:00 AM +0 3 * * 0 cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --weekly >> logs/backup.log 2>&1 + +# Backup Health Check: Daily at 8:00 AM +0 8 * * * cd /opt/ferrero-automation/Python-Version && ./database/check_backups.sh --quiet >> logs/backup_check.log 2>&1 +``` + +Save and exit. + +#### Step 4.4: Monitor Initial Runs + +```bash +# Watch logs in real-time +tail -f logs/cron_a1_a2.log + +# In another terminal, watch all workflow logs +tail -f logs/*.log + +# Monitor for 1-2 hours +# Look for: +# - ✅ "No A1 campaigns found" (normal if no campaigns ready) +# - ✅ Successful processing messages +# - ⚠️ Any errors or authentication failures +``` + +#### Step 4.5: Verify First Campaign Processing + +**When first A1 campaign appears:** + +```bash +# Monitor A1→A2 processing +tail -f logs/a1_to_a2.log + +# Expected flow: +# - Campaign detected +# - Assets downloaded from DAM +# - Uploaded to Box with tracking IDs +# - Stored in database +# - Status updated A1→A2 +# - Email sent + +# Verify in database +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT tracking_id, original_filename, created_at FROM master_assets ORDER BY created_at DESC LIMIT 10; +" + +# Verify in Box folder +# Check production Box folder contains files with tracking IDs +``` + +--- + +## Post-Cutover Monitoring + +### First 24 Hours + +#### Hourly Checks: + +```bash +# Check cron execution +tail -100 logs/cron_*.log + +# Check for errors +grep -i error logs/*.log | grep "$(date +%Y-%m-%d)" + +# Check database growth +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT + (SELECT COUNT(*) FROM master_assets) as masters, + (SELECT COUNT(*) FROM derivative_assets) as derivatives, + (SELECT COUNT(*) FROM creativex_scores) as creativex, + (SELECT COUNT(*) FROM campaign_status) as campaigns; +" + +# Check Box folders +# Verify files are being created/deleted as expected +``` + +#### Email Monitoring: + +- ✅ Success emails arriving +- ⚠️ No error emails (or expected errors only) +- 📊 Daily report at 7 PM contains production data + +#### Workflow Health: + +```bash +# Check all workflows completed recent runs +ls -lt logs/*.log | head -10 + +# Verify campaigns progressing through stages +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT campaign_number, status, webhook_sent, updated_at +FROM campaign_status +ORDER BY updated_at DESC +LIMIT 10; +" +``` + +### First Week + +#### Daily Checks: + +1. **Review daily report email** (7 PM) + - Verify statistics are accurate + - Check success rates + - Review any errors + +2. **Check backup health** + ```bash + ./database/check_backups.sh + ``` + +3. **Monitor disk space** + ```bash + df -h /opt/ferrero-automation + du -sh Python-Version/backups/ + du -sh Python-Version/logs/ + du -sh Python-Version/temp/ + ``` + +4. **Check cron logs** + ```bash + # Verify cron is executing + grep CRON /var/log/syslog | grep ferrero | tail -20 + ``` + +#### Weekly Tasks: + +- [ ] Review workflow success rates +- [ ] Check for any recurring errors +- [ ] Verify webhook delivery to Make.com +- [ ] Test restore from backup (monthly, but good to test early) +- [ ] Review with Ferrero team + +--- + +## Rollback Plan + +### If Critical Issues Found + +#### Quick Rollback (Revert to Staging) + +```bash +# 1. Stop production cron jobs +crontab -e +# Comment out all lines + +# 2. Restore staging .env +cp .env.staging.backup .env + +# 3. Restore staging Box-config.json +cp ../Box-config.json.staging.backup ../Box-config.json + +# 4. Update environment +nano .env +# Change: ENV=production → ENV=staging + +# 5. Restart staging cron jobs +crontab -e +# Restore original cron jobs +``` + +#### Database Rollback + +```bash +# If database needs rollback +./database/restore.sh backups/dumps/.sql.gz +``` + +#### Code Rollback + +```bash +# If code changes need rollback +git log --oneline | head -10 # Find commit before cutover +git revert +git push origin main +``` + +--- + +## Known Production Differences + +### Development vs Production + +| Component | Development | Production | Notes | +|-----------|-------------|------------|-------| +| **DAM URL** | ppr.dam.ferrero.com | dam.ferrero.com (TBD) | Confirm with Ferrero IT | +| **Box Folders** | Dev folder IDs | Prod folder IDs | Create new folders | +| **OAuth2 Creds** | Dev credentials | Prod credentials | Request from Ferrero | +| **Email Recipients** | daveporter@oliver.agency | Ferrero team + ops | Update recipient lists | +| **Webhook URL** | Dev Make scenario | Prod Make scenario | Create new scenario | +| **DB Password** | ferrero_pass_2025 | Strong production password | Change for security | +| **mTLS Cert** | dev-mtls-dev.pfx | dam-mtls-prod.pfx | Get prod certificate | +| **Server IP** | Dev IP | Prod IP | Whitelist for mTLS | + +--- + +## Cutover Communication Plan + +### Stakeholders to Notify + +**Before Cutover (48 hours):** +- [ ] Ferrero DAM team +- [ ] Ferrero IT team +- [ ] Oliver Agency operations +- [ ] Content team users + +**Email Template:** +``` +Subject: Ferrero DAM Automation - Production Cutover [Date] + +The automated content scaling system will be going live on [DATE] at [TIME]. + +What's changing: +- Automation moves from staging to production DAM environment +- Campaigns will automatically progress through workflow stages +- Email notifications will come from production system + +What to expect: +- A1→A2: Master assets auto-downloaded to Box +- A2→A3: Localized assets auto-uploaded to DAM +- A5→A6: Rejected assets flagged for rework +- CreativeX scores automatically attached to assets + +Support during cutover: +- [Contact person]: [Email/Phone] +- Monitoring: First 24 hours continuous +- Rollback available if issues found + +Please report any issues immediately to [ERROR_EMAIL] +``` + +**During Cutover:** +- [ ] Status updates every 2 hours +- [ ] Available for immediate response + +**After Cutover (24 hours):** +- [ ] Summary email with statistics +- [ ] Known issues (if any) +- [ ] Next steps + +--- + +## Post-Cutover Validation + +### Validation Checklist (First 24 Hours) + +#### Workflow Validations: + +**A1→A2 (Master Download):** +- [ ] Campaign detected with status A1 +- [ ] All assets downloaded from DAM +- [ ] Uploaded to Box with tracking IDs +- [ ] Stored in database with full metadata +- [ ] Status updated to A2 +- [ ] Success email sent +- [ ] Tracking IDs are unique +- [ ] Subfolder structure preserved + +**A5→A6 (Rejections/Rework):** +- [ ] Only "NOT APPROVED" assets downloaded +- [ ] Rejection comments extracted +- [ ] Tracking IDs reused correctly +- [ ] Email lists rejection reasons +- [ ] Approved assets skipped + +**B1→B2 (Global Masters):** +- [ ] Only "Global comm" campaigns processed +- [ ] Uploaded to separate Box folder +- [ ] MASTERS_ prefix in folder names +- [ ] Email sent (no webhook) + +**A2→A3 (Agency Uploads):** +- [ ] Files detected in Box A2→A3 folder +- [ ] Tracking IDs parsed from filenames +- [ ] Master metadata loaded from database +- [ ] CreativeX scores looked up from database +- [ ] Default values used when scores missing +- [ ] Uploaded to correct DAM folder +- [ ] Files deleted from Box after success +- [ ] Email shows CreativeX status + +**CreativeX Scoring:** +- [ ] PDFs processed from Box folder 350605024645 +- [ ] Scores extracted via LlamaExtract +- [ ] Stored in database with version tracking +- [ ] Decimals removed from IDs and scores +- [ ] PDFs deleted after successful extraction +- [ ] Email sent with version badges + +#### Database Validations: + +```bash +# Check all tables have data +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT + 'master_assets' as table_name, COUNT(*) as records FROM master_assets +UNION ALL SELECT + 'derivative_assets', COUNT(*) FROM derivative_assets +UNION ALL SELECT + 'creativex_scores', COUNT(*) FROM creativex_scores +UNION ALL SELECT + 'campaign_status', COUNT(*) FROM campaign_status +UNION ALL SELECT + 'asset_events', COUNT(*) FROM asset_events; +" + +# Check for any errors in logs +grep -i error logs/*.log | grep "$(date +%Y-%m-%d)" +``` + +#### Backup Validations: + +```bash +# Verify first production backup +./database/check_backups.sh + +# Expected: +# - Backup created at 2:00 AM +# - Backup file size reasonable +# - No errors in backup.log +``` + +--- + +## Production Support Procedures + +### Monitoring Dashboard Commands + +**Create monitoring script:** `scripts/production_status.sh` + +```bash +#!/bin/bash +echo "=========================================" +echo "Ferrero Automation Production Status" +echo "=========================================" +echo "" + +# Database stats +echo "Database Records:" +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT + (SELECT COUNT(*) FROM master_assets WHERE status = 'active') as active_masters, + (SELECT COUNT(*) FROM creativex_scores WHERE status = 'active') as active_scores, + (SELECT COUNT(*) FROM campaign_status WHERE webhook_sent = true) as completed_campaigns; +" + +echo "" +echo "Recent Activity (Last 24 hours):" +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT COUNT(*) as new_assets_today +FROM master_assets +WHERE created_at > NOW() - INTERVAL '24 hours'; +" + +echo "" +echo "Backup Status:" +./database/check_backups.sh --quiet + +echo "" +echo "Recent Errors:" +grep -i error logs/*.log | grep "$(date +%Y-%m-%d)" | tail -5 +``` + +### Emergency Contacts + +**Escalation Path:** +1. **Level 1:** Operations team (monitors daily) +2. **Level 2:** Technical lead (Dave Porter) +3. **Level 3:** Ferrero IT team + +**Contact Information:** +- Operations: operations@oliver.agency +- Technical Lead: daveporter@oliver.agency +- Ferrero IT: [TO BE PROVIDED] + +--- + +## Configuration Checklist Summary + +### Files to Update: + +- [ ] `Python-Version/.env` - All production credentials +- [ ] `../Box-config.json` - Production Box JWT +- [ ] `config/certificates/dam-mtls-prod.pfx` - Production mTLS cert (if using) +- [ ] Crontab - Production schedule +- [ ] `/etc/hosts` - Production hostname if needed + +### Credentials to Obtain: + +- [ ] Production DAM OAuth2 client ID and secret +- [ ] Production DAM base URL and auth URL +- [ ] Production Box folder IDs (4 folders) +- [ ] Production Box JWT credentials +- [ ] Production mTLS certificate (if using) +- [ ] Production webhook URL (Make.com) +- [ ] Production email recipients list +- [ ] Production LlamaCloud API key (or confirm dev key works) + +### Testing to Complete: + +- [ ] DAM connection test +- [ ] Box connection test +- [ ] Database connection test +- [ ] Email delivery test +- [ ] Webhook delivery test +- [ ] CreativeX extraction test +- [ ] End-to-end workflow test with small campaign +- [ ] Backup and restore test + +--- + +## Success Metrics + +### Week 1 Targets: + +- **Uptime:** > 99% (workflows running without crashes) +- **Success Rate:** > 95% (assets processed successfully) +- **Email Delivery:** 100% (all notifications received) +- **Backup Success:** 7/7 daily backups completed +- **Response Time:** < 10 minutes (cron cycle) + +### Monitor These KPIs: + +```bash +# Check success rate from daily report emails +# Or query database: +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " +SELECT + COUNT(*) FILTER (WHERE webhook_sent = true) as successful_campaigns, + COUNT(*) as total_campaigns, + ROUND(100.0 * COUNT(*) FILTER (WHERE webhook_sent = true) / NULLIF(COUNT(*), 0), 1) as success_rate +FROM campaign_status +WHERE created_at > NOW() - INTERVAL '7 days'; +" +``` + +--- + +## Troubleshooting Production Issues + +### Issue: Workflows Not Running + +```bash +# Check cron service +systemctl status cron + +# Check cron logs +grep CRON /var/log/syslog | tail -50 + +# Manually run workflow +cd /opt/ferrero-automation/Python-Version +source venv/bin/activate +python scripts/a1_to_a2_download.py +``` + +### Issue: Authentication Failing + +```bash +# Test OAuth2 +python scripts/test_connection.py + +# Test mTLS +python scripts/test_connection.py --auth-pfx + +# Check credentials in .env +grep "DAM_" .env | grep -v PASSWORD +``` + +### Issue: Database Not Accessible + +```bash +# Check PostgreSQL container +docker ps | grep ferrero-tracking-db + +# Check logs +docker logs ferrero-tracking-db | tail -50 + +# Test connection +PGPASSWORD= psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT 1;" +``` + +### Issue: Box Upload/Download Failing + +```bash +# Test Box connection +python -c " +from shared.config_loader import load_config +from shared.box_client import BoxClient +config = load_config('config/config.yaml') +box = BoxClient(config) +print(box.test_connection()) +" + +# Check Box-config.json location +ls -la ../Box-config.json + +# Verify Box folder IDs +grep "BOX_ROOT_FOLDER" .env +``` + +### Issue: Emails Not Sending + +```bash +# Check SMTP settings +grep "SMTP" .env + +# Test email manually +python -c " +from shared.config_loader import load_config +from shared.notifier import Notifier +config = load_config('config/config.yaml') +notifier = Notifier(config) +# Send test email... +" + +# Check Mailgun logs (if using Mailgun) +``` + +--- + +## Appendix: Production Readiness Checklist + +### Infrastructure: + +- [ ] Production server provisioned and accessible +- [ ] Docker installed and running +- [ ] PostgreSQL container running on port 5437 +- [ ] Sufficient disk space (minimum 10 GB) +- [ ] Network access to DAM, Box, email servers +- [ ] Firewall rules configured (if using mTLS) +- [ ] SSL certificates valid (if applicable) + +### Code Deployment: + +- [ ] Latest code pulled from Bitbucket +- [ ] Virtual environment created (Python 3.10+) +- [ ] All dependencies installed (requirements.txt) +- [ ] llama-cloud-services installed +- [ ] Scripts executable (chmod +x) +- [ ] Database schema initialized (init.sql) +- [ ] creativex_scores table created + +### Configuration: + +- [ ] .env file configured with production values +- [ ] Box-config.json in correct location (../Box-config.json) +- [ ] mTLS certificate in place (if using) +- [ ] All folder IDs updated +- [ ] Email recipients updated +- [ ] Webhook URL updated + +### Testing: + +- [ ] All connection tests passed +- [ ] Test campaign processed successfully +- [ ] Email notifications received +- [ ] Webhook delivery confirmed +- [ ] CreativeX extraction working +- [ ] Backup system tested + +### Automation: + +- [ ] Cron jobs configured +- [ ] Cron service running +- [ ] Log rotation configured +- [ ] Backup schedule active +- [ ] Monitoring scripts in place + +### Documentation: + +- [ ] Team trained on system +- [ ] Support procedures documented +- [ ] Rollback plan understood +- [ ] Escalation path defined +- [ ] Emergency contacts listed + +### Sign-off: + +- [ ] Technical lead approval: _________________ Date: _______ +- [ ] Operations approval: _________________ Date: _______ +- [ ] Ferrero stakeholder approval: _________________ Date: _______ + +--- + +## Post-Cutover: Continuous Improvement + +### Month 1 Review: + +- Analyze success rates +- Identify recurring errors +- Optimize cron schedule if needed +- Tune email notifications +- Adjust backup retention if needed + +### Potential Enhancements: + +- [ ] Add Grafana/Prometheus monitoring +- [ ] Implement alerting thresholds +- [ ] Add Slack notifications +- [ ] Remote backup to S3/cloud storage +- [ ] Point-in-time recovery (WAL archiving) +- [ ] High availability setup (if needed) +- [ ] Load balancing (if volume increases) + +--- + +**Document Version:** 1.0 +**Last Updated:** November 11, 2025 +**Owner:** Dave Porter (daveporter@oliver.agency) +**Review Date:** [After cutover + 1 week]