DJP 20a187a61d feat: Add Python project dependencies.

2025-11-20 22:45:04 -05:00

32 KiB

Raw Permalink Blame History

Ferrero Automation - Production Cutover Plan

Overview

This document outlines the complete cutover process from development/staging to production environment for the Ferrero DAM Content Scaling automation system.

Current State: Running on staging/dev environment Target State: Full production deployment with live DAM integration

Pre-Cutover Checklist

1. Verify Current System Health

Development Environment:

# Check all workflows are working
python scripts/a1_to_a2_download.py
python scripts/a5_to_a6_download.py
python scripts/b1_to_b2_download.py
python scripts/a2_to_a3_upload_polling.py
python scripts/creativex_scoring_storing.py

# Verify database connectivity
python scripts/test_connection.py

# Check recent logs for errors
grep -i error logs/*.log | tail -50

Expected Results:

✅ All scripts execute without errors
✅ Database connections successful
✅ Box authentication working
✅ DAM OAuth2 authentication working
✅ Email notifications sending

2. Backup Current Configuration

# Backup current .env file
cp .env .env.backup.$(date +%Y%m%d)

# Backup database
./database/backup.sh --daily

# Backup Box-config.json
cp ../Box-config.json ../Box-config.json.backup.$(date +%Y%m%d)

# Backup current cron jobs
crontab -l > cron_backup_$(date +%Y%m%d).txt

3. Document Current URLs and IDs

Create reference document with current settings:

# Extract current settings
grep "^DAM_BASE_URL" .env
grep "^BOX_ROOT_FOLDER" .env
grep "WEBHOOK" .env

Production Credentials & Configuration

Required Production Values

1. DAM Production Endpoints

Current (Dev/Staging):

DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token

Production:

DAM_BASE_URL=https://dam.ferrero.com/otmmapi  # VERIFY actual production URL
DAM_AUTH_URL=https://dam.ferrero.com/otdsws/oauth2/token  # VERIFY

ACTION REQUIRED:

Confirm production DAM URLs with Ferrero IT team
Obtain production OAuth2 credentials
Test authentication against production endpoint

2. DAM OAuth2 Credentials

Current (Dev):

DAM_CLIENT_ID=otds-OLV
DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1

Production:

DAM_CLIENT_ID=<production_client_id>
DAM_CLIENT_SECRET=<production_client_secret>

ACTION REQUIRED:

Request production OAuth2 client credentials from Ferrero
Test credentials against production DAM
Verify token refresh works

3. Box Folder IDs

Current (Dev/Staging):

BOX_ROOT_FOLDER_A1_A2=348304357505      # Local master downloads
BOX_ROOT_FOLDER_A2_A3=348526703108      # Agency upload processing
BOX_ROOT_FOLDER_B1_B2=349261192115      # Global master downloads
BOX_ROOT_FOLDER_CREATIVEX=350605024645  # CreativeX PDF scoring

Production:

BOX_ROOT_FOLDER_A1_A2=<production_folder_id>
BOX_ROOT_FOLDER_A2_A3=<production_folder_id>
BOX_ROOT_FOLDER_B1_B2=<production_folder_id>
BOX_ROOT_FOLDER_CREATIVEX=<production_folder_id>

ACTION REQUIRED:

Create production Box folders with correct structure
Document folder IDs
Set proper permissions for Box service account
Test upload/download to each folder

4. Box Authentication

Current (Dev):

BOX_CLIENT_ID=l2atwxxq4xna7phcjr2uifm4mbah69qp
BOX_CLIENT_SECRET=6XcuCQ6akpk9daE0UHaGSv3mSxWaER4l
BOX_JWT_KEY_ID=n1izyn3l
BOX_PASSPHRASE=971585f5fd6171428c14a7c8899af5ab
BOX_ENTERPRISE_ID=43984435

Production:

BOX_CLIENT_ID=<production_value>
BOX_CLIENT_SECRET=<production_value>
BOX_JWT_KEY_ID=<production_value>
BOX_PASSPHRASE=<production_value>
BOX_ENTERPRISE_ID=<production_value>

File: ../Box-config.json (one level up from Python-Version)

ACTION REQUIRED:

Request production Box JWT credentials
Generate new Box-config.json for production
Place at correct location (../Box-config.json)
Set permissions: chmod 600 ../Box-config.json
Test Box authentication

5. Database

Current (Dev):

DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=ferrero_pass_2025

Production Options:

Option A: Keep local PostgreSQL (recommended for start)

DB_HOST=localhost
DB_PORT=5437
DB_USER=ferrero_user
DB_PASSWORD=<new_strong_production_password>

Option B: Use managed database service

DB_HOST=<rds_or_managed_db_hostname>
DB_PORT=5432
DB_USER=ferrero_prod_user
DB_PASSWORD=<production_password>

ACTION REQUIRED:

Decide: Local PostgreSQL or managed service
Change default password (ferrero_pass_2025)
Update .env with production credentials
Initialize production database with init.sql
Set up automated backups (cron jobs)

6. Email Notifications

Current (Dev):

SMTP_SERVER=smtp.mailgun.org
SMTP_USER=twist@mail.dev.oliver.solutions
SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency
ERROR_EMAIL=daveporter@oliver.agency
REPORT_EMAILS=daveporter@oliver.agency

Production:

SMTP_SERVER=smtp.mailgun.org
SMTP_USER=<production_mailgun_user>
SENDER_EMAIL=ferrero-automation@ferrero.com  # Or oliver.agency
ERROR_EMAIL=ferrero-it@ferrero.com,daveporter@oliver.agency
REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency

ACTION REQUIRED:

Set up production email domain
Update recipient lists for production team
Test email delivery
Verify spam filters don't block automated emails

7. Webhooks

Current (Dev):

CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/3f9ztwl8qnljufo0l65utfv5wvvnt9m5

Production:

CAMPAIGN_STATUS_WEBHOOK_URL=<production_make_scenario_url>

ACTION REQUIRED:

Create production Make.com scenario
Update webhook URL
Test webhook delivery
Verify webhook signature validation (if enabled)

8. CreativeX / LlamaCloud

Current (Dev):

LLAMA_CLOUD_API_KEY=llx-EDmfh0ZReUbXUbaa5i5275TAP2LznNDqc3skJRL3HY4RUDcf
CREATIVEX_AGENT_NAME=Creativex-Extract

Production:

LLAMA_CLOUD_API_KEY=<production_api_key>
CREATIVEX_AGENT_NAME=Creativex-Extract  # Same agent or production-specific?

ACTION REQUIRED:

Confirm same LlamaCloud API key for production or get new one
Verify agent "Creativex-Extract" exists and works in production
Test extraction with production data

9. mTLS Certificate (Optional)

Current (Dev):

DAM_MTLS_BASE_URL=https://dev-auth.app-api.ferrero.com/00003/mm
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-dev.pfx
DAM_MTLS_CERT_PASSWORD=fnJ8xrnh!54NE&2HR62=2P3YEy+hy9RajZ7v5&=y

Production:

DAM_MTLS_BASE_URL=https://prod-auth.app-api.ferrero.com/00003/mm  # VERIFY
DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx
DAM_MTLS_CERT_PASSWORD=<production_cert_password>

ACTION REQUIRED:

Obtain production mTLS certificate
Place in config/certificates/
Set permissions: chmod 600 config/certificates/*.pfx
Whitelist production server IP
Test mTLS authentication: python scripts/test_connection.py --auth-pfx

Cutover Steps

Phase 1: Preparation (1-2 days before)

Step 1.1: Update Production Server Code

# SSH to production server
ssh user@production-server

# Navigate to deployment directory
cd /opt/ferrero-automation/Python-Version

# Pull latest code from Bitbucket
git pull origin main

# Verify all new files present
ls -la scripts/creativex_scoring_storing.py
ls -la database/backup.sh
ls -la DATABASE_BACKUP_GUIDE.md

Step 1.2: Install Dependencies

# Activate virtual environment
source venv/bin/activate

# Install new dependencies (llama-cloud-services)
sudo venv/bin/pip install llama-cloud-services

# Verify installation
pip show llama-cloud-services

# Or reinstall all dependencies
sudo venv/bin/pip install -r requirements.txt

Step 1.3: Database Updates

# Create creativex_scores table
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql

# Or just add the new table
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
CREATE TABLE IF NOT EXISTS creativex_scores (
    id SERIAL PRIMARY KEY,
    filename VARCHAR(500) NOT NULL,
    box_file_id VARCHAR(255),
    creativex_id VARCHAR(255),
    creativex_url TEXT,
    quality_score VARCHAR(50),
    full_extraction_data JSONB,
    extracted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    status VARCHAR(50) DEFAULT 'active',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_creativex_filename ON creativex_scores(filename);
CREATE INDEX IF NOT EXISTS idx_creativex_box_file ON creativex_scores(box_file_id);
CREATE INDEX IF NOT EXISTS idx_creativex_status ON creativex_scores(status);
"

# Verify table created
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores"

Step 1.4: Update .env with Production Credentials

# Backup current .env
cp .env .env.staging.backup

# Edit .env with production values
nano .env

Update these sections:

# DAM Production URLs
DAM_BASE_URL=<production_dam_url>
DAM_AUTH_URL=<production_auth_url>
DAM_CLIENT_ID=<production_client_id>
DAM_CLIENT_SECRET=<production_client_secret>

# Box Production Folder IDs
BOX_ROOT_FOLDER_A1_A2=<production_folder>
BOX_ROOT_FOLDER_A2_A3=<production_folder>
BOX_ROOT_FOLDER_B1_B2=<production_folder>
BOX_ROOT_FOLDER_CREATIVEX=<production_folder>

# Database Production Password
DB_PASSWORD=<new_strong_password>

# Email Production Recipients
ERROR_EMAIL=ferrero-it@ferrero.com,operations@oliver.agency
REPORT_EMAILS=ferrero-team@ferrero.com,operations@oliver.agency

# Webhooks Production URL
CAMPAIGN_STATUS_WEBHOOK_URL=<production_webhook>

# CreativeX Production
LLAMA_CLOUD_API_KEY=<production_key>

Save and exit.

Step 1.5: Update Box-config.json

# Backup current Box config
cp ../Box-config.json ../Box-config.json.staging.backup

# Replace with production Box JWT config
nano ../Box-config.json

Paste production Box JWT credentials (JSON format from Box Developer Console).

# Set secure permissions
chmod 600 ../Box-config.json

Step 1.6: Add mTLS Production Certificate (If Using)

# Copy production certificate
cp /path/to/dam-mtls-prod.pfx config/certificates/

# Set permissions
chmod 600 config/certificates/dam-mtls-prod.pfx

# Update .env
nano .env
# Update: DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx
# Update: DAM_MTLS_CERT_PASSWORD=<production_password>

Phase 2: Testing (Before Cutover)

Step 2.1: Test Database Connection

# Test PostgreSQL
PGPASSWORD=<new_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"

# Test via Python
python scripts/test_connection.py

Expected: Connection successful, tables accessible

Step 2.2: Test DAM Authentication

# Test OAuth2 (production credentials)
python scripts/test_connection.py

# Test mTLS (if using)
python scripts/test_connection.py --auth-pfx

Expected: DAM connection successful, can retrieve campaigns

Step 2.3: Test Box Authentication

# Run Box connection test
python -c "
from shared.config_loader import load_config
from shared.box_client import BoxClient
config = load_config('config/config.yaml')
box = BoxClient(config, root_folder_id=config['box']['root_folder_a1_a2'])
print('Box test:', box.test_connection())
"

Expected: Box connection OK

Step 2.4: Test Email Notifications

# Send test email
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
notifier.send_email(
    template_name='a1_to_a2_no_assets',
    recipients=config['notifications']['recipients']['success'],
    data={
        'campaign_name': 'TEST CAMPAIGN',
        'campaign_id': 'TEST123',
        'campaign_number': 'C000000999'
    }
)
print('Test email sent')
"

Expected: Email received by production recipients

Step 2.5: Test Webhook

# Test webhook delivery
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
result = notifier.send_webhook(
    url=config['webhooks']['campaign_status_update']['url'],
    payload={'test': 'cutover_test', 'campaign': 'TEST'}
)
print('Webhook test:', result)
"

Expected: Webhook received in production Make.com scenario

Step 2.6: Test CreativeX Extraction

# Upload test PDF to production CreativeX folder
# Then run extraction
python scripts/creativex_scoring_storing.py

# Verify in database
PGPASSWORD=<new_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT filename, quality_score FROM creativex_scores WHERE status = 'active' LIMIT 5;
"

Expected: Scores extracted and stored successfully

Step 2.7: Dry Run Workflows (Production Data, No Status Updates)

Important: Test with real production campaigns but DON'T update statuses yet

# Test A1→A2 download (will process but not update status without all assets)
# Manually select a small test campaign first
python scripts/a1_to_a2_download.py

# Check logs
tail -100 logs/a1_to_a2.log

# Verify files in Box production folder
# Verify records in database

# Test A2→A3 upload with test file
# Upload one file with tracking ID to A2→A3 folder
python scripts/a2_to_a3_upload_polling.py

# Verify upload to DAM
# Check email notification

CRITICAL: During dry run testing:

✅ Use real production endpoints
✅ Use real production Box folders
⚠️ Process SMALL test campaigns only
⚠️ Verify you're comfortable with status updates before going live
⚠️ Have rollback plan ready

Phase 3: Go-Live Decision Point

Go/No-Go Criteria

✅ GO if all true:

All connection tests passed
Email notifications working
Webhooks delivering successfully
Test campaign processed without errors
Box uploads/downloads working
DAM uploads working with correct metadata
CreativeX scores extracting and integrating
Database backups configured and tested
Team trained on monitoring/support
Rollback plan documented

⛔ NO-GO if any true:

Authentication failing
Production credentials incomplete
Email delivery issues
Webhook not reaching Make.com
Test uploads corrupted or failed
Metadata mapping incorrect
No backup system in place

Go-Live Execution

Phase 4: Cutover Day

Step 4.1: Final Preparation (Morning)

# 1. Stop all cron jobs temporarily
crontab -e
# Comment out all Ferrero automation lines (add # at start)

# 2. Create final backup of staging data
./database/backup.sh --daily

# 3. Verify no workflows are running
ps aux | grep python | grep scripts

# 4. Clear any pending Box files
# Manually verify Box folders are empty or in known state

Step 4.2: Update Environment Variable

# Update environment setting
nano .env

# Change:
ENV=staging
# To:
ENV=production

# Save and exit

Step 4.3: Enable Production Cron Jobs

crontab -e

Production Cron Schedule:

# A1→A2: Download master assets from DAM (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron_a1_a2.log 2>&1

# A5→A6: Download rejected assets for rework (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a5_to_a6_download.py >> logs/cron_a5_a6.log 2>&1

# B1→B2: Download global master campaigns (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/b1_to_b2_download.py >> logs/cron_b1_b2.log 2>&1

# A2→A3: Process agency uploads from Box (every 5 minutes)
*/5 * * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/a2_to_a3_upload_polling.py >> logs/cron_a2_a3.log 2>&1

# Daily Report: Send summary email (7:00 PM daily)
0 19 * * * cd /opt/ferrero-automation/Python-Version && venv/bin/python scripts/daily_report.py >> logs/daily_report.log 2>&1

# Database Backup: Daily at 2:00 AM
0 2 * * * cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --daily >> logs/backup.log 2>&1

# Database Backup: Weekly on Sundays at 3:00 AM
0 3 * * 0 cd /opt/ferrero-automation/Python-Version && ./database/backup.sh --weekly >> logs/backup.log 2>&1

# Backup Health Check: Daily at 8:00 AM
0 8 * * * cd /opt/ferrero-automation/Python-Version && ./database/check_backups.sh --quiet >> logs/backup_check.log 2>&1

Save and exit.

Step 4.4: Monitor Initial Runs

# Watch logs in real-time
tail -f logs/cron_a1_a2.log

# In another terminal, watch all workflow logs
tail -f logs/*.log

# Monitor for 1-2 hours
# Look for:
# - ✅ "No A1 campaigns found" (normal if no campaigns ready)
# - ✅ Successful processing messages
# - ⚠️ Any errors or authentication failures

Step 4.5: Verify First Campaign Processing

When first A1 campaign appears:

# Monitor A1→A2 processing
tail -f logs/a1_to_a2.log

# Expected flow:
# - Campaign detected
# - Assets downloaded from DAM
# - Uploaded to Box with tracking IDs
# - Stored in database
# - Status updated A1→A2
# - Email sent

# Verify in database
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT tracking_id, original_filename, created_at FROM master_assets ORDER BY created_at DESC LIMIT 10;
"

# Verify in Box folder
# Check production Box folder contains files with tracking IDs

Post-Cutover Monitoring

First 24 Hours

Hourly Checks:

# Check cron execution
tail -100 logs/cron_*.log

# Check for errors
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"

# Check database growth
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    (SELECT COUNT(*) FROM master_assets) as masters,
    (SELECT COUNT(*) FROM derivative_assets) as derivatives,
    (SELECT COUNT(*) FROM creativex_scores) as creativex,
    (SELECT COUNT(*) FROM campaign_status) as campaigns;
"

# Check Box folders
# Verify files are being created/deleted as expected

Email Monitoring:

✅ Success emails arriving
⚠️ No error emails (or expected errors only)
📊 Daily report at 7 PM contains production data

Workflow Health:

# Check all workflows completed recent runs
ls -lt logs/*.log | head -10

# Verify campaigns progressing through stages
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT campaign_number, status, webhook_sent, updated_at
FROM campaign_status
ORDER BY updated_at DESC
LIMIT 10;
"

First Week

Daily Checks:

Review daily report email (7 PM)
- Verify statistics are accurate
- Check success rates
- Review any errors
Check backup health
```
./database/check_backups.sh
```

Monitor disk space

df -h /opt/ferrero-automation
du -sh Python-Version/backups/
du -sh Python-Version/logs/
du -sh Python-Version/temp/

Check cron logs

# Verify cron is executing
grep CRON /var/log/syslog | grep ferrero | tail -20

Weekly Tasks:

Review workflow success rates
Check for any recurring errors
Verify webhook delivery to Make.com
Test restore from backup (monthly, but good to test early)
Review with Ferrero team

Rollback Plan

If Critical Issues Found

Quick Rollback (Revert to Staging)

# 1. Stop production cron jobs
crontab -e
# Comment out all lines

# 2. Restore staging .env
cp .env.staging.backup .env

# 3. Restore staging Box-config.json
cp ../Box-config.json.staging.backup ../Box-config.json

# 4. Update environment
nano .env
# Change: ENV=production → ENV=staging

# 5. Restart staging cron jobs
crontab -e
# Restore original cron jobs

Database Rollback

# If database needs rollback
./database/restore.sh backups/dumps/<pre_cutover_backup>.sql.gz

Code Rollback

# If code changes need rollback
git log --oneline | head -10  # Find commit before cutover
git revert <commit_hash>
git push origin main

Known Production Differences

Development vs Production

Component	Development	Production	Notes
DAM URL	ppr.dam.ferrero.com	dam.ferrero.com (TBD)	Confirm with Ferrero IT
Box Folders	Dev folder IDs	Prod folder IDs	Create new folders
OAuth2 Creds	Dev credentials	Prod credentials	Request from Ferrero
Email Recipients	daveporter@oliver.agency	Ferrero team + ops	Update recipient lists
Webhook URL	Dev Make scenario	Prod Make scenario	Create new scenario
DB Password	ferrero_pass_2025	Strong production password	Change for security
mTLS Cert	dev-mtls-dev.pfx	dam-mtls-prod.pfx	Get prod certificate
Server IP	Dev IP	Prod IP	Whitelist for mTLS

Cutover Communication Plan

Stakeholders to Notify

Before Cutover (48 hours):

Ferrero DAM team
Ferrero IT team
Oliver Agency operations
Content team users

Email Template:

Subject: Ferrero DAM Automation - Production Cutover [Date]

The automated content scaling system will be going live on [DATE] at [TIME].

What's changing:
- Automation moves from staging to production DAM environment
- Campaigns will automatically progress through workflow stages
- Email notifications will come from production system

What to expect:
- A1→A2: Master assets auto-downloaded to Box
- A2→A3: Localized assets auto-uploaded to DAM
- A5→A6: Rejected assets flagged for rework
- CreativeX scores automatically attached to assets

Support during cutover:
- [Contact person]: [Email/Phone]
- Monitoring: First 24 hours continuous
- Rollback available if issues found

Please report any issues immediately to [ERROR_EMAIL]

During Cutover:

Status updates every 2 hours
Available for immediate response

After Cutover (24 hours):

Summary email with statistics
Known issues (if any)
Next steps

Post-Cutover Validation

Validation Checklist (First 24 Hours)

Workflow Validations:

A1→A2 (Master Download):

Campaign detected with status A1
All assets downloaded from DAM
Uploaded to Box with tracking IDs
Stored in database with full metadata
Status updated to A2
Success email sent
Tracking IDs are unique
Subfolder structure preserved

A5→A6 (Rejections/Rework):

Only "NOT APPROVED" assets downloaded
Rejection comments extracted
Tracking IDs reused correctly
Email lists rejection reasons
Approved assets skipped

B1→B2 (Global Masters):

Only "Global comm" campaigns processed
Uploaded to separate Box folder
MASTERS_ prefix in folder names
Email sent (no webhook)

A2→A3 (Agency Uploads):

Files detected in Box A2→A3 folder
Tracking IDs parsed from filenames
Master metadata loaded from database
CreativeX scores looked up from database
Default values used when scores missing
Uploaded to correct DAM folder
Files deleted from Box after success
Email shows CreativeX status

CreativeX Scoring:

PDFs processed from Box folder 350605024645
Scores extracted via LlamaExtract
Stored in database with version tracking
Decimals removed from IDs and scores
PDFs deleted after successful extraction
Email sent with version badges

Database Validations:

# Check all tables have data
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    'master_assets' as table_name, COUNT(*) as records FROM master_assets
UNION ALL SELECT
    'derivative_assets', COUNT(*) FROM derivative_assets
UNION ALL SELECT
    'creativex_scores', COUNT(*) FROM creativex_scores
UNION ALL SELECT
    'campaign_status', COUNT(*) FROM campaign_status
UNION ALL SELECT
    'asset_events', COUNT(*) FROM asset_events;
"

# Check for any errors in logs
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"

Backup Validations:

# Verify first production backup
./database/check_backups.sh

# Expected:
# - Backup created at 2:00 AM
# - Backup file size reasonable
# - No errors in backup.log

Production Support Procedures

Monitoring Dashboard Commands

Create monitoring script: scripts/production_status.sh

#!/bin/bash
echo "========================================="
echo "Ferrero Automation Production Status"
echo "========================================="
echo ""

# Database stats
echo "Database Records:"
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    (SELECT COUNT(*) FROM master_assets WHERE status = 'active') as active_masters,
    (SELECT COUNT(*) FROM creativex_scores WHERE status = 'active') as active_scores,
    (SELECT COUNT(*) FROM campaign_status WHERE webhook_sent = true) as completed_campaigns;
"

echo ""
echo "Recent Activity (Last 24 hours):"
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT COUNT(*) as new_assets_today
FROM master_assets
WHERE created_at > NOW() - INTERVAL '24 hours';
"

echo ""
echo "Backup Status:"
./database/check_backups.sh --quiet

echo ""
echo "Recent Errors:"
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)" | tail -5

Emergency Contacts

Escalation Path:

Level 1: Operations team (monitors daily)
Level 2: Technical lead (Dave Porter)
Level 3: Ferrero IT team

Contact Information:

Operations: operations@oliver.agency
Technical Lead: daveporter@oliver.agency
Ferrero IT: [TO BE PROVIDED]

Configuration Checklist Summary

Files to Update:

Python-Version/.env - All production credentials
../Box-config.json - Production Box JWT
config/certificates/dam-mtls-prod.pfx - Production mTLS cert (if using)
Crontab - Production schedule
/etc/hosts - Production hostname if needed

Credentials to Obtain:

Production DAM OAuth2 client ID and secret
Production DAM base URL and auth URL
Production Box folder IDs (4 folders)
Production Box JWT credentials
Production mTLS certificate (if using)
Production webhook URL (Make.com)
Production email recipients list
Production LlamaCloud API key (or confirm dev key works)

Testing to Complete:

DAM connection test
Box connection test
Database connection test
Email delivery test
Webhook delivery test
CreativeX extraction test
End-to-end workflow test with small campaign
Backup and restore test

Success Metrics

Week 1 Targets:

Uptime: > 99% (workflows running without crashes)
Success Rate: > 95% (assets processed successfully)
Email Delivery: 100% (all notifications received)
Backup Success: 7/7 daily backups completed
Response Time: < 10 minutes (cron cycle)

Monitor These KPIs:

# Check success rate from daily report emails
# Or query database:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT
    COUNT(*) FILTER (WHERE webhook_sent = true) as successful_campaigns,
    COUNT(*) as total_campaigns,
    ROUND(100.0 * COUNT(*) FILTER (WHERE webhook_sent = true) / NULLIF(COUNT(*), 0), 1) as success_rate
FROM campaign_status
WHERE created_at > NOW() - INTERVAL '7 days';
"

Troubleshooting Production Issues

Issue: Workflows Not Running

# Check cron service
systemctl status cron

# Check cron logs
grep CRON /var/log/syslog | tail -50

# Manually run workflow
cd /opt/ferrero-automation/Python-Version
source venv/bin/activate
python scripts/a1_to_a2_download.py

Issue: Authentication Failing

# Test OAuth2
python scripts/test_connection.py

# Test mTLS
python scripts/test_connection.py --auth-pfx

# Check credentials in .env
grep "DAM_" .env | grep -v PASSWORD

Issue: Database Not Accessible

# Check PostgreSQL container
docker ps | grep ferrero-tracking-db

# Check logs
docker logs ferrero-tracking-db | tail -50

# Test connection
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT 1;"

Issue: Box Upload/Download Failing

# Test Box connection
python -c "
from shared.config_loader import load_config
from shared.box_client import BoxClient
config = load_config('config/config.yaml')
box = BoxClient(config)
print(box.test_connection())
"

# Check Box-config.json location
ls -la ../Box-config.json

# Verify Box folder IDs
grep "BOX_ROOT_FOLDER" .env

Issue: Emails Not Sending

# Check SMTP settings
grep "SMTP" .env

# Test email manually
python -c "
from shared.config_loader import load_config
from shared.notifier import Notifier
config = load_config('config/config.yaml')
notifier = Notifier(config)
# Send test email...
"

# Check Mailgun logs (if using Mailgun)

Appendix: Production Readiness Checklist

Infrastructure:

Production server provisioned and accessible
Docker installed and running
PostgreSQL container running on port 5437
Sufficient disk space (minimum 10 GB)
Network access to DAM, Box, email servers
Firewall rules configured (if using mTLS)
SSL certificates valid (if applicable)

Code Deployment:

Latest code pulled from Bitbucket
Virtual environment created (Python 3.10+)
All dependencies installed (requirements.txt)
llama-cloud-services installed
Scripts executable (chmod +x)
Database schema initialized (init.sql)
creativex_scores table created

Configuration:

.env file configured with production values
Box-config.json in correct location (../Box-config.json)
mTLS certificate in place (if using)
All folder IDs updated
Email recipients updated
Webhook URL updated

Testing:

All connection tests passed
Test campaign processed successfully
Email notifications received
Webhook delivery confirmed
CreativeX extraction working
Backup system tested

Automation:

Cron jobs configured
Cron service running
Log rotation configured
Backup schedule active
Monitoring scripts in place

Documentation:

Team trained on system
Support procedures documented
Rollback plan understood
Escalation path defined
Emergency contacts listed

Sign-off:

Technical lead approval: _________________ Date: _______
Operations approval: _________________ Date: _______
Ferrero stakeholder approval: _________________ Date: _______

Post-Cutover: Continuous Improvement

Month 1 Review:

Analyze success rates
Identify recurring errors
Optimize cron schedule if needed
Tune email notifications
Adjust backup retention if needed

Potential Enhancements:

Add Grafana/Prometheus monitoring
Implement alerting thresholds
Add Slack notifications
Remote backup to S3/cloud storage
Point-in-time recovery (WAL archiving)
High availability setup (if needed)
Load balancing (if volume increases)

Document Version: 1.0 Last Updated: November 11, 2025 Owner: Dave Porter (daveporter@oliver.agency) Review Date: [After cutover + 1 week]

32 KiB Raw Permalink Blame History

Ferrero Automation - Production Cutover Plan

Overview

Pre-Cutover Checklist

1. Verify Current System Health

2. Backup Current Configuration

3. Document Current URLs and IDs

Production Credentials & Configuration

Required Production Values

1. DAM Production Endpoints

2. DAM OAuth2 Credentials

3. Box Folder IDs

4. Box Authentication

5. Database

6. Email Notifications

7. Webhooks

8. CreativeX / LlamaCloud

9. mTLS Certificate (Optional)

Cutover Steps

Phase 1: Preparation (1-2 days before)

Step 1.1: Update Production Server Code

Step 1.2: Install Dependencies

Step 1.3: Database Updates

Step 1.4: Update .env with Production Credentials

Step 1.5: Update Box-config.json

Step 1.6: Add mTLS Production Certificate (If Using)

Phase 2: Testing (Before Cutover)

Step 2.1: Test Database Connection

Step 2.2: Test DAM Authentication

Step 2.3: Test Box Authentication

Step 2.4: Test Email Notifications

Step 2.5: Test Webhook

Step 2.6: Test CreativeX Extraction

Step 2.7: Dry Run Workflows (Production Data, No Status Updates)

Phase 3: Go-Live Decision Point

Go/No-Go Criteria

Go-Live Execution

Phase 4: Cutover Day

Step 4.1: Final Preparation (Morning)

Step 4.2: Update Environment Variable

Step 4.3: Enable Production Cron Jobs

Step 4.4: Monitor Initial Runs

Step 4.5: Verify First Campaign Processing

Post-Cutover Monitoring

First 24 Hours

Hourly Checks:

Email Monitoring:

Workflow Health:

First Week

Daily Checks:

Weekly Tasks:

Rollback Plan

If Critical Issues Found

Quick Rollback (Revert to Staging)

Database Rollback

Code Rollback

Known Production Differences

Development vs Production

Cutover Communication Plan

Stakeholders to Notify

Post-Cutover Validation

Validation Checklist (First 24 Hours)

Workflow Validations:

Database Validations:

Backup Validations:

Production Support Procedures

Monitoring Dashboard Commands

Emergency Contacts

Configuration Checklist Summary

Files to Update:

Credentials to Obtain:

Testing to Complete:

Success Metrics

Week 1 Targets:

Monitor These KPIs:

Troubleshooting Production Issues

Issue: Workflows Not Running

Issue: Authentication Failing

Issue: Database Not Accessible

Issue: Box Upload/Download Failing

32 KiB

Raw Permalink Blame History