ferrero-opentext/Python-Version/MARKDOWN_DOCS/CUTOVER-TODOS.md

27 KiB

Ferrero Automation - Production Cutover TODO List

Status: Not Started Target Go-Live Date: _____________ Completed By: _____________


PRE-CUTOVER PHASE

1. Obtain Production Credentials

DAM Production

  • Confirm production DAM base URL with Ferrero IT
    • Current (staging): https://ppr.dam.ferrero.com/otmmapi
    • Production: __________________________
  • Confirm production DAM auth URL
    • Current (staging): https://ppr.dam.ferrero.com/otdsws/oauth2/token
    • Production: __________________________
  • Obtain production OAuth2 client ID
    • Production value: __________________________
  • Obtain production OAuth2 client secret
    • Production value: __________________________
  • Test DAM authentication against production endpoint
    • Test command: python scripts/test_connection.py
    • Result: /

Box Production

  • Create production Box folder for A1→A2 (Local master downloads)
    • Folder ID: __________________________
    • Folder name: Ferrero DAM - A1→A2 Local Masters
  • Create production Box folder for A2→A3 (Agency uploads)
    • Folder ID: __________________________
    • Folder name: Ferrero DAM - A2→A3 Agency Processing
  • Create production Box folder for B1→B2 (Global masters)
    • Folder ID: __________________________
    • Folder name: Ferrero DAM - B1→B2 Global Masters
  • Create production Box folder for CreativeX PDFs
    • Folder ID: __________________________
    • Folder name: Ferrero DAM - CreativeX Scoring
  • Obtain production Box JWT credentials
    • Client ID: __________________________
    • Client Secret: __________________________
    • JWT Key ID: __________________________
    • Enterprise ID: __________________________
  • Generate production Box-config.json file
    • File location: ../Box-config.json
    • Permissions set: chmod 600 ../Box-config.json
  • Test Box authentication
    • Test result: /

Database Production

  • Decide: Local PostgreSQL or managed database service
    • Decision: __________________________
  • Generate strong production database password
    • Password stored in: __________________________
  • If managed DB: Obtain connection details
    • Host: __________________________
    • Port: __________________________
    • Username: __________________________

Email Production

  • Set up production Mailgun/SMTP account (or confirm dev account works)
    • SMTP User: __________________________
    • Sender Email: __________________________
  • Define production email recipients
    • Error notifications: __________________________
    • Success reports: __________________________
    • Daily reports: __________________________
  • Test email delivery to production recipients
    • Test result: /

Webhooks Production

  • Create production Make.com scenario
    • Scenario URL: __________________________
  • Configure webhook authentication (if needed)
    • Auth token: __________________________
  • Test webhook delivery
    • Test result: /

CreativeX Production

  • Confirm LlamaCloud API key for production
    • Use dev key: Yes / No
    • Production key: __________________________
  • Verify agent "Creativex-Extract" exists in production
    • Agent verified: /
  • Test CreativeX extraction with production credentials
    • Test result: /

mTLS Certificate (Optional)

  • Obtain production mTLS certificate (.pfx file)
    • Certificate location: config/certificates/dam-mtls-prod.pfx
  • Obtain certificate password
    • Password: __________________________
  • Whitelist production server IP with Ferrero IT
    • Server IP: __________________________
    • Whitelisted: /
  • Test mTLS authentication
    • Test command: python scripts/test_connection.py --auth-pfx
    • Result: /

2. Backup Current System

  • Backup current .env file
    • Command: cp .env .env.staging.backup.$(date +%Y%m%d)
    • Backup location: __________________________
  • Backup current Box-config.json
    • Command: cp ../Box-config.json ../Box-config.json.staging.backup
    • Backup location: __________________________
  • Backup current database
    • Command: ./database/backup.sh --daily
    • Backup file: __________________________
  • Backup current cron jobs
    • Command: crontab -l > cron_staging_backup_$(date +%Y%m%d).txt
    • Backup file: __________________________
  • Document current configuration
    • DAM URLs: __________________________
    • Box folder IDs: __________________________
    • Database settings: __________________________

3. Prepare Production Server

  • SSH access to production server confirmed
    • Server hostname: __________________________
    • Username: __________________________
  • Git repository cloned to /opt/ferrero-automation/Python-Version
    • Path verified: /
  • Python 3.10+ installed on server
    • Python version: __________________________
  • Docker installed and running
    • Docker version: __________________________
  • PostgreSQL container running on port 5437
    • Container status: /
  • Sufficient disk space available
    • Available space: __________ GB
    • Required minimum: 10 GB

INSTALLATION PHASE

4. Deploy Code to Production Server

  • SSH to production server
    • Command: ssh user@production-server
  • Navigate to deployment directory
    • Command: cd /opt/ferrero-automation/Python-Version
  • Pull latest code from Bitbucket
    • Command: git pull origin main
    • Latest commit: __________________________
  • Verify new files present
    • scripts/creativex_scoring_storing.py: /
    • database/backup.sh: /
    • database/restore.sh: /
    • database/check_backups.sh: /
    • CUTOVER.md: /

5. Install Dependencies

  • Create virtual environment (if not exists)
    • Command: python3 -m venv venv
  • Activate virtual environment
    • Command: source venv/bin/activate
  • Install Python dependencies
    • Command: pip install -r requirements.txt
    • Or: sudo venv/bin/pip install -r requirements.txt
  • Install llama-cloud-services specifically
    • Command: sudo venv/bin/pip install llama-cloud-services
  • Verify installations
    • Command: pip list | grep -E "boxsdk|psycopg2|llama-cloud"
    • boxsdk installed: /
    • psycopg2-binary installed: /
    • llama-cloud-services installed: /

6. Configure Production Environment

  • Update .env file with production credentials

    • Command: nano .env
  • Set environment to production

    • ENV=production /
  • Update DAM URLs and credentials

    • DAM_BASE_URL=<production> /
    • DAM_AUTH_URL=<production> /
    • DAM_CLIENT_ID=<production> /
    • DAM_CLIENT_SECRET=<production> /
  • Update Box folder IDs

    • BOX_ROOT_FOLDER_A1_A2=<production> /
    • BOX_ROOT_FOLDER_A2_A3=<production> /
    • BOX_ROOT_FOLDER_B1_B2=<production> /
    • BOX_ROOT_FOLDER_CREATIVEX=<production> /
  • Update database password

    • DB_PASSWORD=<strong_production_password> /
  • Update email recipients

    • ERROR_EMAIL=<production_emails> /
    • REPORT_EMAILS=<production_emails> /
  • Update webhook URL

    • CAMPAIGN_STATUS_WEBHOOK_URL=<production> /
  • Update CreativeX credentials

    • LLAMA_CLOUD_API_KEY=<production> /
  • Update mTLS settings (if using)

    • DAM_MTLS_BASE_URL=<production> /
    • DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx /
    • DAM_MTLS_CERT_PASSWORD=<production> /
  • Save .env file

  • Set secure permissions on .env

    • Command: chmod 600 .env
  • Update Box-config.json with production credentials

    • Command: nano ../Box-config.json
    • Paste production JWT JSON
    • Save and set permissions: chmod 600 ../Box-config.json
  • Copy production mTLS certificate (if using)

    • Command: cp /path/to/cert config/certificates/dam-mtls-prod.pfx
    • Set permissions: chmod 600 config/certificates/*.pfx

7. Initialize Production Database

  • Verify PostgreSQL container running
    • Command: docker ps | grep ferrero-tracking-db
    • Container running: /
  • Create database schema
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql
  • Verify all tables created
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\dt"
    • Expected tables:
      • master_assets /
      • derivative_assets /
      • creativex_scores /
      • campaign_status /
      • asset_events /
      • workflow_state /
  • Verify creativex_scores table structure
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores"
    • Has 10 columns: /
    • Has 3 indexes: /

8. Make Scripts Executable

  • Set executable permissions on workflow scripts
    • Command: chmod +x scripts/*.py
  • Set executable permissions on backup scripts
    • Command: chmod +x database/*.sh
  • Verify permissions
    • Command: ls -la scripts/*.py database/*.sh

TESTING PHASE

9. Test All Connections

  • Test database connection

    • Command: python scripts/test_connection.py
    • Result: /
    • Error (if any): __________________________
  • Test DAM OAuth2 authentication

    • Command: python scripts/test_connection.py
    • Result: /
    • Error (if any): __________________________
  • Test DAM mTLS authentication (if using)

    • Command: python scripts/test_connection.py --auth-pfx
    • Result: /
    • Error (if any): __________________________
  • Test Box authentication

    • Command: python -c "from shared.config_loader import load_config; from shared.box_client import BoxClient; config = load_config('config/config.yaml'); box = BoxClient(config); print('Box:', box.test_connection())"
    • Result: /
    • Error (if any): __________________________

10. Test Email Notifications

  • Send test email to production recipients
    • Command: See CUTOVER.md "Step 2.4"
    • Email received by ERROR_EMAIL: /
    • Email received by REPORT_EMAILS: /
    • Email formatting correct: /

11. Test Webhook Delivery

  • Send test webhook to production Make.com
    • Command: See CUTOVER.md "Step 2.5"
    • Webhook received: /
    • Make.com scenario triggered: /

12. Test CreativeX Extraction

  • Upload test PDF to production CreativeX Box folder
    • PDF uploaded: /
    • Folder ID: __________________________
  • Run CreativeX scoring script
    • Command: python scripts/creativex_scoring_storing.py
    • Extraction successful: /
    • Score stored in database: /
    • PDF deleted from Box: /
    • Email sent: /
  • Verify in database
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT * FROM creativex_scores LIMIT 1;"
    • Record found: /

13. Dry Run Workflows (IMPORTANT: Use Small Test Campaign)

A1→A2 Test (Master Download)

  • Identify small test campaign in production DAM with status A1
    • Campaign ID: __________________________
    • Campaign name: __________________________
    • Number of assets: __________
  • Run A1→A2 script manually
    • Command: python scripts/a1_to_a2_download.py
    • Assets downloaded: /
    • Uploaded to Box: /
    • Tracking IDs generated: /
    • Stored in database: /
    • Status updated A1→A2: /
    • Email sent: /
  • Verify files in production Box folder
    • Files present with tracking IDs: /
  • Verify database records
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets ORDER BY created_at DESC LIMIT 5;"
    • Records found: /

A2→A3 Test (Agency Upload)

  • Upload test file to production A2→A3 Box folder
    • Filename format: Brand_Country_Language_123_OMG999_<tracking_id>.mp4
    • Tracking ID matches master asset: /
  • Run A2→A3 script manually
    • Command: python scripts/a2_to_a3_upload_polling.py
    • File downloaded from Box: /
    • Master metadata loaded: /
    • CreativeX score looked up: /
    • Uploaded to DAM: /
    • File deleted from Box: /
    • Email sent: /
  • Verify upload in production DAM
    • Asset ID: __________________________
    • Metadata correct: /
    • CreativeX score attached: /
  • Check email notification
    • Shows CreativeX status: /
    • Default values if score missing: /

A5→A6 Test (Rejections)

  • Identify test campaign with rejected assets (status A5)
    • Campaign ID: __________________________
    • Has "NOT APPROVED" assets: /
  • Run A5→A6 script
    • Command: python scripts/a5_to_a6_download.py
    • Only rejected assets downloaded: /
    • Tracking IDs reused: /
    • Rejection comments extracted: /
    • Email shows rejection details: /

B1→B2 Test (Global Masters)

  • Identify test global campaign (status B1)
    • Campaign ID: __________________________
    • Campaign type: "Global comm" /
  • Run B1→B2 script
    • Command: python scripts/b1_to_b2_download.py
    • Downloaded to separate Box folder: /
    • MASTERS_ prefix used: /
    • Email sent (no webhook): /

14. Configure Backup System

  • Make backup scripts executable
    • Command: chmod +x database/*.sh
  • Create backup directories
    • Command: mkdir -p backups/{dumps,basebackups,safety,manual}
  • Set backup directory permissions
    • Command: chmod 700 backups
  • Run test backup
    • Command: ./database/backup.sh --daily
    • Backup created: /
    • Backup file: __________________________
  • Verify backup integrity
    • Command: gunzip -t backups/dumps/<latest>.sql.gz
    • Valid: /
  • Run backup health check
    • Command: ./database/check_backups.sh
    • Status: /

GO/NO-GO DECISION

15. Pre-Go-Live Validation

Review all tests completed above. GO LIVE only if ALL criteria met:

  • All production credentials obtained and tested
  • All connection tests passed (DAM, Box, Database, Email, Webhook)
  • Test campaign processed successfully through A1→A2
  • Test file uploaded successfully through A2→A3
  • CreativeX extraction working
  • Emails delivering to production recipients
  • Webhook triggering Make.com scenario
  • Database backups configured and tested
  • Rollback plan documented and understood
  • Team trained and ready to monitor
  • Emergency contacts documented

GO/NO-GO Decision:

  • GO - Proceed with cutover
  • NO-GO - Delay cutover, address issues above

Decision Made By: _____________ Date: _____________ Time: _____________


GO-LIVE PHASE

16. Cutover Execution (Go-Live Day)

Morning Preparation

  • Announce cutover to stakeholders
    • Email sent: /
    • Recipients acknowledged: /
  • Stop any running staging workflows
    • Staging cron disabled: /
  • Create final pre-cutover backup
    • Command: ./database/backup.sh --daily
    • Backup file: __________________________
  • Verify production server ready
    • All services running: /
    • Disk space available: /
    • Network connectivity: /

Enable Production Automation

  • Configure production cron jobs
    • Command: crontab -e
    • Add all cron jobs from CUTOVER.md
  • Cron jobs configured:
    • A1→A2 every 5 minutes: /
    • A5→A6 every 5 minutes: /
    • B1→B2 every 5 minutes: /
    • A2→A3 every 5 minutes: /
    • Daily report at 7 PM: /
    • Daily backup at 2 AM: /
    • Weekly backup Sundays 3 AM: /
    • Backup health check 8 AM: /
  • Save crontab
  • Verify cron service running
    • Command: systemctl status cron
    • Status: /

Initial Monitoring (First 2 Hours)

  • Watch workflow logs
    • Command: tail -f logs/*.log
    • Monitoring start time: __________________________
  • Verify first cron execution (wait 5 minutes)
    • A1→A2 executed: /
    • A5→A6 executed: /
    • B1→B2 executed: /
    • A2→A3 executed: /
  • Check for errors in first runs
    • Command: grep -i error logs/cron_*.log
    • No errors found: /
    • Errors (if any): __________________________
  • Verify "No campaigns found" is normal
    • Normal if no A1/A5/B1 campaigns ready: /

17. First Campaign Processing (Production)

When first production campaign reaches A1 status:

  • Campaign detected in logs
    • Campaign ID: __________________________
    • Campaign name: __________________________
    • Detected at: __________________________
  • Monitor A1→A2 processing
    • Command: tail -f logs/a1_to_a2.log
  • Verify each step:
    • Assets downloaded from DAM: /
    • Uploaded to Box with tracking IDs: /
    • Stored in database: /
    • Status updated A1→A2: /
    • Email sent: /
  • Check database records
    • Command: PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets WHERE local_campaign_id = '<campaign_number>';"
    • All assets recorded: /
  • Verify files in Box
    • Box folder ID: __________________________
    • All files present: /
    • Filenames have tracking IDs: /

18. Agency Upload Testing (A2→A3)

When agency uploads first localized file:

  • File detected in A2→A3 folder
    • Filename: __________________________
    • Tracking ID parsed: __________________________
  • Monitor processing
    • Command: tail -f logs/a2_to_a3.log
  • Verify steps:
    • Downloaded from Box: /
    • Master metadata loaded: /
    • CreativeX score found in database: / / N/A
    • CreativeX default used if missing: / N/A
    • Uploaded to DAM: /
    • Deleted from Box: /
    • Email sent: /
  • Check email notification
    • CreativeX status shown: /
    • Warning if score missing: / N/A

19. CreativeX Score Integration

  • Upload CreativeX PDF to production folder
    • PDF uploaded: /
    • Folder ID: __________________________
  • Run scoring script (manual or wait for cron if configured)
    • Command: python scripts/creativex_scoring_storing.py
    • Score extracted: /
    • Stored in database: /
    • PDF deleted: /
    • Email sent: /
  • Verify A2→A3 picks up score
    • Upload matching filename to A2→A3
    • Score attached to DAM upload: /
    • Email shows score found: /

POST-CUTOVER MONITORING

20. First 24 Hours - Hourly Checks

Hour 1-2 (Intensive Monitoring):

  • Hour 1: Check all workflow logs for errors
    • Command: grep -i error logs/cron_*.log
    • No critical errors: /
  • Hour 2: Verify cron executions
    • Command: grep CRON /var/log/syslog | grep ferrero | tail -20
    • All workflows executing: /

Hour 3-6 (Regular Monitoring):

  • Hour 3: Check database growth
    • Command: See CUTOVER.md monitoring section
    • Records increasing: /
  • Hour 4: Check Box folders
    • Files being created/deleted: /
  • Hour 5: Verify emails sending
    • Emails arriving: /
  • Hour 6: Check for any stuck workflows
    • No stuck processes: /

Hour 7-24 (Light Monitoring):

  • Hour 12: Review any errors
    • Error count: __________
    • Critical errors: None / Found
  • Hour 18: Check disk space
    • Command: df -h /opt/ferrero-automation
    • Space available: __________ GB
  • Hour 24: Review day 1 summary
    • Campaigns processed: __________
    • Assets processed: __________
    • Success rate: __________%

21. Daily Monitoring (First Week)

Day 1

  • Review daily report email (7 PM)
    • Email received: /
    • Statistics accurate: /
    • Success rate: __________%
    • Errors: __________________________
  • Check backup completed
    • Command: ./database/check_backups.sh
    • Backup at 2 AM: /
    • Backup file size: __________
  • Review workflow logs
    • A1→A2 campaigns: __________
    • A2→A3 uploads: __________
    • A5→A6 rejections: __________
    • Errors: __________________________
  • Check database records
    • Total master assets: __________
    • Total CreativeX scores: __________
    • Total campaigns: __________

Day 2

  • Daily report reviewed
    • Success rate: __________%
  • Backup health check
    • Last backup age: __________ hours
  • Log review for errors
    • Critical errors: __________
  • Stakeholder update sent
    • Email sent: /

Day 3

  • Daily report reviewed
    • Success rate: __________%
  • Backup verified
    • 3 daily backups present: /
  • CreativeX integration working
    • Scores being used in A2→A3: /

Day 4

  • Daily report reviewed
    • Success rate: __________%
  • Mid-week status update to team
    • Update sent: /
  • Check temp directory cleanup
    • Command: du -sh temp/
    • Size reasonable: /

Day 5

  • Daily report reviewed
    • Success rate: __________%
  • Review any recurring errors
    • Recurring issues: __________________________
  • Check cron logs for patterns
    • All workflows executing: /

Day 6

  • Daily report reviewed
    • Success rate: __________%
  • Prepare for weekend monitoring
    • On-call person identified: __________________________

Day 7

  • Daily report reviewed
    • Success rate: __________%
  • Verify Sunday weekly backup
    • Backup at 3 AM Sunday: /
    • Basebackup file created: /
  • Week 1 summary report
    • Total campaigns: __________
    • Total assets: __________
    • Overall success rate: __________%
    • Issues resolved: __________
    • Outstanding issues: __________________________

VALIDATION & SIGN-OFF

22. Week 1 Validation

  • All workflows running successfully
    • A1→A2: /
    • A5→A6: /
    • B1→B2: /
    • A2→A3: /
    • CreativeX: /
  • Email notifications working
    • Success emails: /
    • Error emails: /
    • Daily reports: /
  • Webhooks delivering
    • Make.com receiving: /
  • Backups completing daily
    • 7 daily backups: /
    • 1 weekly backup: /
  • Database healthy
    • No corruption: /
    • Performance good: /
  • No critical errors
    • Error rate < 5%: /
  • Stakeholders satisfied
    • Ferrero approval: /
    • Oliver approval: /

23. Production Sign-Off

  • Technical validation complete

    • Signed: _________________ Date: _____________
    • Name: Dave Porter
    • Role: Technical Lead
  • Operations validation complete

    • Signed: _________________ Date: _____________
    • Name: __________________________
    • Role: Operations Manager
  • Business validation complete

    • Signed: _________________ Date: _____________
    • Name: __________________________
    • Role: Ferrero Stakeholder

ROLLBACK PROCEDURES (If Needed)

24. Emergency Rollback

Only if critical production issues found:

  • Stop production cron jobs immediately

    • Command: crontab -e (comment out all lines)
    • Cron stopped: /
  • Restore staging .env

    • Command: cp .env.staging.backup .env
    • ENV=staging set: /
  • Restore staging Box-config.json

    • Command: cp ../Box-config.json.staging.backup ../Box-config.json
    • Staging config restored: /
  • Restore database (if needed)

    • Command: ./database/restore.sh <pre_cutover_backup>
    • Database restored: /
  • Notify stakeholders of rollback

    • Email sent: /
    • Reason documented: __________________________
  • Document what went wrong

    • Issue description: __________________________
    • Root cause: __________________________
    • Resolution plan: __________________________
  • Resume staging cron jobs

    • Staging cron enabled: /

COMPLETION

25. Final Steps

  • Update documentation with production details
    • Production URLs documented: /
    • Production folder IDs documented: /
  • Archive staging backups
    • Staging backups moved to archive: /
  • Update README.md with production status
    • Status updated: /
  • Schedule Week 2 review meeting
    • Meeting scheduled: /
    • Date: __________________________
  • Close cutover project
    • Cutover complete: /
    • Date: __________________________
    • Final notes: __________________________

NOTES & ISSUES LOG

Issues Encountered During Cutover

Issue 1:

  • Description: __________________________
  • Severity: High / Medium / Low
  • Resolution: __________________________
  • Resolved by: __________________________
  • Date/Time: __________________________

Issue 2:

  • Description: __________________________
  • Severity: High / Medium / Low
  • Resolution: __________________________
  • Resolved by: __________________________
  • Date/Time: __________________________

Issue 3:

  • Description: __________________________
  • Severity: High / Medium / Low
  • Resolution: __________________________
  • Resolved by: __________________________
  • Date/Time: __________________________

Lessons Learned

What went well:

  1. __________________________
  2. __________________________
  3. __________________________

What could be improved:

  1. __________________________
  2. __________________________
  3. __________________________

Recommendations for future:

  1. __________________________
  2. __________________________
  3. __________________________

QUICK REFERENCE

Important Commands

# Check system status
./database/check_backups.sh
python scripts/test_connection.py
docker ps | grep ferrero

# View logs
tail -f logs/*.log
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"

# Database queries
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"

# Manual workflow execution
python scripts/a1_to_a2_download.py
python scripts/a2_to_a3_upload_polling.py
python scripts/creativex_scoring_storing.py

# Backup and restore
./database/backup.sh --daily
./database/restore.sh --list

Emergency Contacts

Critical File Locations

  • Configuration: /opt/ferrero-automation/Python-Version/.env
  • Box JWT: /opt/ferrero-automation/Box-config.json
  • Logs: /opt/ferrero-automation/Python-Version/logs/
  • Backups: /opt/ferrero-automation/Python-Version/backups/
  • Database: Docker container ferrero-tracking-db port 5437

Cutover Status: Not Started | 🟡 In Progress | Complete | Rolled Back

Current Status: __________________________

Cutover Completed: _____________ By: _____________

Production Verified: _____________ By: _____________