27 KiB
Ferrero Automation - Production Cutover TODO List
Status: Not Started Target Go-Live Date: _____________ Completed By: _____________
PRE-CUTOVER PHASE
1. Obtain Production Credentials
DAM Production
- Confirm production DAM base URL with Ferrero IT
- Current (staging):
https://ppr.dam.ferrero.com/otmmapi - Production:
__________________________
- Current (staging):
- Confirm production DAM auth URL
- Current (staging):
https://ppr.dam.ferrero.com/otdsws/oauth2/token - Production:
__________________________
- Current (staging):
- Obtain production OAuth2 client ID
- Production value:
__________________________
- Production value:
- Obtain production OAuth2 client secret
- Production value:
__________________________
- Production value:
- Test DAM authentication against production endpoint
- Test command:
python scripts/test_connection.py - Result: ✅ / ❌
- Test command:
Box Production
- Create production Box folder for A1→A2 (Local master downloads)
- Folder ID:
__________________________ - Folder name:
Ferrero DAM - A1→A2 Local Masters
- Folder ID:
- Create production Box folder for A2→A3 (Agency uploads)
- Folder ID:
__________________________ - Folder name:
Ferrero DAM - A2→A3 Agency Processing
- Folder ID:
- Create production Box folder for B1→B2 (Global masters)
- Folder ID:
__________________________ - Folder name:
Ferrero DAM - B1→B2 Global Masters
- Folder ID:
- Create production Box folder for CreativeX PDFs
- Folder ID:
__________________________ - Folder name:
Ferrero DAM - CreativeX Scoring
- Folder ID:
- Obtain production Box JWT credentials
- Client ID:
__________________________ - Client Secret:
__________________________ - JWT Key ID:
__________________________ - Enterprise ID:
__________________________
- Client ID:
- Generate production Box-config.json file
- File location:
../Box-config.json - Permissions set:
chmod 600 ../Box-config.json
- File location:
- Test Box authentication
- Test result: ✅ / ❌
Database Production
- Decide: Local PostgreSQL or managed database service
- Decision:
__________________________
- Decision:
- Generate strong production database password
- Password stored in:
__________________________
- Password stored in:
- If managed DB: Obtain connection details
- Host:
__________________________ - Port:
__________________________ - Username:
__________________________
- Host:
Email Production
- Set up production Mailgun/SMTP account (or confirm dev account works)
- SMTP User:
__________________________ - Sender Email:
__________________________
- SMTP User:
- Define production email recipients
- Error notifications:
__________________________ - Success reports:
__________________________ - Daily reports:
__________________________
- Error notifications:
- Test email delivery to production recipients
- Test result: ✅ / ❌
Webhooks Production
- Create production Make.com scenario
- Scenario URL:
__________________________
- Scenario URL:
- Configure webhook authentication (if needed)
- Auth token:
__________________________
- Auth token:
- Test webhook delivery
- Test result: ✅ / ❌
CreativeX Production
- Confirm LlamaCloud API key for production
- Use dev key: Yes / No
- Production key:
__________________________
- Verify agent "Creativex-Extract" exists in production
- Agent verified: ✅ / ❌
- Test CreativeX extraction with production credentials
- Test result: ✅ / ❌
mTLS Certificate (Optional)
- Obtain production mTLS certificate (.pfx file)
- Certificate location:
config/certificates/dam-mtls-prod.pfx
- Certificate location:
- Obtain certificate password
- Password:
__________________________
- Password:
- Whitelist production server IP with Ferrero IT
- Server IP:
__________________________ - Whitelisted: ✅ / ❌
- Server IP:
- Test mTLS authentication
- Test command:
python scripts/test_connection.py --auth-pfx - Result: ✅ / ❌
- Test command:
2. Backup Current System
- Backup current .env file
- Command:
cp .env .env.staging.backup.$(date +%Y%m%d) - Backup location:
__________________________
- Command:
- Backup current Box-config.json
- Command:
cp ../Box-config.json ../Box-config.json.staging.backup - Backup location:
__________________________
- Command:
- Backup current database
- Command:
./database/backup.sh --daily - Backup file:
__________________________
- Command:
- Backup current cron jobs
- Command:
crontab -l > cron_staging_backup_$(date +%Y%m%d).txt - Backup file:
__________________________
- Command:
- Document current configuration
- DAM URLs:
__________________________ - Box folder IDs:
__________________________ - Database settings:
__________________________
- DAM URLs:
3. Prepare Production Server
- SSH access to production server confirmed
- Server hostname:
__________________________ - Username:
__________________________
- Server hostname:
- Git repository cloned to
/opt/ferrero-automation/Python-Version- Path verified: ✅ / ❌
- Python 3.10+ installed on server
- Python version:
__________________________
- Python version:
- Docker installed and running
- Docker version:
__________________________
- Docker version:
- PostgreSQL container running on port 5437
- Container status: ✅ / ❌
- Sufficient disk space available
- Available space:
__________ GB - Required minimum: 10 GB
- Available space:
INSTALLATION PHASE
4. Deploy Code to Production Server
- SSH to production server
- Command:
ssh user@production-server
- Command:
- Navigate to deployment directory
- Command:
cd /opt/ferrero-automation/Python-Version
- Command:
- Pull latest code from Bitbucket
- Command:
git pull origin main - Latest commit:
__________________________
- Command:
- Verify new files present
scripts/creativex_scoring_storing.py: ✅ / ❌database/backup.sh: ✅ / ❌database/restore.sh: ✅ / ❌database/check_backups.sh: ✅ / ❌CUTOVER.md: ✅ / ❌
5. Install Dependencies
- Create virtual environment (if not exists)
- Command:
python3 -m venv venv
- Command:
- Activate virtual environment
- Command:
source venv/bin/activate
- Command:
- Install Python dependencies
- Command:
pip install -r requirements.txt - Or:
sudo venv/bin/pip install -r requirements.txt
- Command:
- Install llama-cloud-services specifically
- Command:
sudo venv/bin/pip install llama-cloud-services
- Command:
- Verify installations
- Command:
pip list | grep -E "boxsdk|psycopg2|llama-cloud" - boxsdk installed: ✅ / ❌
- psycopg2-binary installed: ✅ / ❌
- llama-cloud-services installed: ✅ / ❌
- Command:
6. Configure Production Environment
-
Update .env file with production credentials
- Command:
nano .env
- Command:
-
Set environment to production
ENV=production✅ / ❌
-
Update DAM URLs and credentials
DAM_BASE_URL=<production>✅ / ❌DAM_AUTH_URL=<production>✅ / ❌DAM_CLIENT_ID=<production>✅ / ❌DAM_CLIENT_SECRET=<production>✅ / ❌
-
Update Box folder IDs
BOX_ROOT_FOLDER_A1_A2=<production>✅ / ❌BOX_ROOT_FOLDER_A2_A3=<production>✅ / ❌BOX_ROOT_FOLDER_B1_B2=<production>✅ / ❌BOX_ROOT_FOLDER_CREATIVEX=<production>✅ / ❌
-
Update database password
DB_PASSWORD=<strong_production_password>✅ / ❌
-
Update email recipients
ERROR_EMAIL=<production_emails>✅ / ❌REPORT_EMAILS=<production_emails>✅ / ❌
-
Update webhook URL
CAMPAIGN_STATUS_WEBHOOK_URL=<production>✅ / ❌
-
Update CreativeX credentials
LLAMA_CLOUD_API_KEY=<production>✅ / ❌
-
Update mTLS settings (if using)
DAM_MTLS_BASE_URL=<production>✅ / ❌DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx✅ / ❌DAM_MTLS_CERT_PASSWORD=<production>✅ / ❌
-
Save .env file
-
Set secure permissions on .env
- Command:
chmod 600 .env
- Command:
-
Update Box-config.json with production credentials
- Command:
nano ../Box-config.json - Paste production JWT JSON
- Save and set permissions:
chmod 600 ../Box-config.json
- Command:
-
Copy production mTLS certificate (if using)
- Command:
cp /path/to/cert config/certificates/dam-mtls-prod.pfx - Set permissions:
chmod 600 config/certificates/*.pfx
- Command:
7. Initialize Production Database
- Verify PostgreSQL container running
- Command:
docker ps | grep ferrero-tracking-db - Container running: ✅ / ❌
- Command:
- Create database schema
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql
- Command:
- Verify all tables created
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\dt" - Expected tables:
master_assets✅ / ❌derivative_assets✅ / ❌creativex_scores✅ / ❌campaign_status✅ / ❌asset_events✅ / ❌workflow_state✅ / ❌
- Command:
- Verify creativex_scores table structure
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores" - Has 10 columns: ✅ / ❌
- Has 3 indexes: ✅ / ❌
- Command:
8. Make Scripts Executable
- Set executable permissions on workflow scripts
- Command:
chmod +x scripts/*.py
- Command:
- Set executable permissions on backup scripts
- Command:
chmod +x database/*.sh
- Command:
- Verify permissions
- Command:
ls -la scripts/*.py database/*.sh
- Command:
TESTING PHASE
9. Test All Connections
-
Test database connection
- Command:
python scripts/test_connection.py - Result: ✅ / ❌
- Error (if any):
__________________________
- Command:
-
Test DAM OAuth2 authentication
- Command:
python scripts/test_connection.py - Result: ✅ / ❌
- Error (if any):
__________________________
- Command:
-
Test DAM mTLS authentication (if using)
- Command:
python scripts/test_connection.py --auth-pfx - Result: ✅ / ❌
- Error (if any):
__________________________
- Command:
-
Test Box authentication
- Command:
python -c "from shared.config_loader import load_config; from shared.box_client import BoxClient; config = load_config('config/config.yaml'); box = BoxClient(config); print('Box:', box.test_connection())" - Result: ✅ / ❌
- Error (if any):
__________________________
- Command:
10. Test Email Notifications
- Send test email to production recipients
- Command: See CUTOVER.md "Step 2.4"
- Email received by ERROR_EMAIL: ✅ / ❌
- Email received by REPORT_EMAILS: ✅ / ❌
- Email formatting correct: ✅ / ❌
11. Test Webhook Delivery
- Send test webhook to production Make.com
- Command: See CUTOVER.md "Step 2.5"
- Webhook received: ✅ / ❌
- Make.com scenario triggered: ✅ / ❌
12. Test CreativeX Extraction
- Upload test PDF to production CreativeX Box folder
- PDF uploaded: ✅ / ❌
- Folder ID:
__________________________
- Run CreativeX scoring script
- Command:
python scripts/creativex_scoring_storing.py - Extraction successful: ✅ / ❌
- Score stored in database: ✅ / ❌
- PDF deleted from Box: ✅ / ❌
- Email sent: ✅ / ❌
- Command:
- Verify in database
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT * FROM creativex_scores LIMIT 1;" - Record found: ✅ / ❌
- Command:
13. Dry Run Workflows (IMPORTANT: Use Small Test Campaign)
A1→A2 Test (Master Download)
- Identify small test campaign in production DAM with status A1
- Campaign ID:
__________________________ - Campaign name:
__________________________ - Number of assets:
__________
- Campaign ID:
- Run A1→A2 script manually
- Command:
python scripts/a1_to_a2_download.py - Assets downloaded: ✅ / ❌
- Uploaded to Box: ✅ / ❌
- Tracking IDs generated: ✅ / ❌
- Stored in database: ✅ / ❌
- Status updated A1→A2: ✅ / ❌
- Email sent: ✅ / ❌
- Command:
- Verify files in production Box folder
- Files present with tracking IDs: ✅ / ❌
- Verify database records
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets ORDER BY created_at DESC LIMIT 5;" - Records found: ✅ / ❌
- Command:
A2→A3 Test (Agency Upload)
- Upload test file to production A2→A3 Box folder
- Filename format:
Brand_Country_Language_123_OMG999_<tracking_id>.mp4 - Tracking ID matches master asset: ✅ / ❌
- Filename format:
- Run A2→A3 script manually
- Command:
python scripts/a2_to_a3_upload_polling.py - File downloaded from Box: ✅ / ❌
- Master metadata loaded: ✅ / ❌
- CreativeX score looked up: ✅ / ❌
- Uploaded to DAM: ✅ / ❌
- File deleted from Box: ✅ / ❌
- Email sent: ✅ / ❌
- Command:
- Verify upload in production DAM
- Asset ID:
__________________________ - Metadata correct: ✅ / ❌
- CreativeX score attached: ✅ / ❌
- Asset ID:
- Check email notification
- Shows CreativeX status: ✅ / ❌
- Default values if score missing: ✅ / ❌
A5→A6 Test (Rejections)
- Identify test campaign with rejected assets (status A5)
- Campaign ID:
__________________________ - Has "NOT APPROVED" assets: ✅ / ❌
- Campaign ID:
- Run A5→A6 script
- Command:
python scripts/a5_to_a6_download.py - Only rejected assets downloaded: ✅ / ❌
- Tracking IDs reused: ✅ / ❌
- Rejection comments extracted: ✅ / ❌
- Email shows rejection details: ✅ / ❌
- Command:
B1→B2 Test (Global Masters)
- Identify test global campaign (status B1)
- Campaign ID:
__________________________ - Campaign type: "Global comm" ✅ / ❌
- Campaign ID:
- Run B1→B2 script
- Command:
python scripts/b1_to_b2_download.py - Downloaded to separate Box folder: ✅ / ❌
- MASTERS_ prefix used: ✅ / ❌
- Email sent (no webhook): ✅ / ❌
- Command:
14. Configure Backup System
- Make backup scripts executable
- Command:
chmod +x database/*.sh
- Command:
- Create backup directories
- Command:
mkdir -p backups/{dumps,basebackups,safety,manual}
- Command:
- Set backup directory permissions
- Command:
chmod 700 backups
- Command:
- Run test backup
- Command:
./database/backup.sh --daily - Backup created: ✅ / ❌
- Backup file:
__________________________
- Command:
- Verify backup integrity
- Command:
gunzip -t backups/dumps/<latest>.sql.gz - Valid: ✅ / ❌
- Command:
- Run backup health check
- Command:
./database/check_backups.sh - Status: ✅ / ❌
- Command:
GO/NO-GO DECISION
15. Pre-Go-Live Validation
Review all tests completed above. GO LIVE only if ALL criteria met:
- All production credentials obtained and tested
- All connection tests passed (DAM, Box, Database, Email, Webhook)
- Test campaign processed successfully through A1→A2
- Test file uploaded successfully through A2→A3
- CreativeX extraction working
- Emails delivering to production recipients
- Webhook triggering Make.com scenario
- Database backups configured and tested
- Rollback plan documented and understood
- Team trained and ready to monitor
- Emergency contacts documented
GO/NO-GO Decision:
- GO - Proceed with cutover
- NO-GO - Delay cutover, address issues above
Decision Made By: _____________ Date: _____________ Time: _____________
GO-LIVE PHASE
16. Cutover Execution (Go-Live Day)
Morning Preparation
- Announce cutover to stakeholders
- Email sent: ✅ / ❌
- Recipients acknowledged: ✅ / ❌
- Stop any running staging workflows
- Staging cron disabled: ✅ / ❌
- Create final pre-cutover backup
- Command:
./database/backup.sh --daily - Backup file:
__________________________
- Command:
- Verify production server ready
- All services running: ✅ / ❌
- Disk space available: ✅ / ❌
- Network connectivity: ✅ / ❌
Enable Production Automation
- Configure production cron jobs
- Command:
crontab -e - Add all cron jobs from CUTOVER.md
- Command:
- Cron jobs configured:
- A1→A2 every 5 minutes: ✅ / ❌
- A5→A6 every 5 minutes: ✅ / ❌
- B1→B2 every 5 minutes: ✅ / ❌
- A2→A3 every 5 minutes: ✅ / ❌
- Daily report at 7 PM: ✅ / ❌
- Daily backup at 2 AM: ✅ / ❌
- Weekly backup Sundays 3 AM: ✅ / ❌
- Backup health check 8 AM: ✅ / ❌
- Save crontab
- Verify cron service running
- Command:
systemctl status cron - Status: ✅ / ❌
- Command:
Initial Monitoring (First 2 Hours)
- Watch workflow logs
- Command:
tail -f logs/*.log - Monitoring start time:
__________________________
- Command:
- Verify first cron execution (wait 5 minutes)
- A1→A2 executed: ✅ / ❌
- A5→A6 executed: ✅ / ❌
- B1→B2 executed: ✅ / ❌
- A2→A3 executed: ✅ / ❌
- Check for errors in first runs
- Command:
grep -i error logs/cron_*.log - No errors found: ✅ / ❌
- Errors (if any):
__________________________
- Command:
- Verify "No campaigns found" is normal
- Normal if no A1/A5/B1 campaigns ready: ✅ / ❌
17. First Campaign Processing (Production)
When first production campaign reaches A1 status:
- Campaign detected in logs
- Campaign ID:
__________________________ - Campaign name:
__________________________ - Detected at:
__________________________
- Campaign ID:
- Monitor A1→A2 processing
- Command:
tail -f logs/a1_to_a2.log
- Command:
- Verify each step:
- Assets downloaded from DAM: ✅ / ❌
- Uploaded to Box with tracking IDs: ✅ / ❌
- Stored in database: ✅ / ❌
- Status updated A1→A2: ✅ / ❌
- Email sent: ✅ / ❌
- Check database records
- Command:
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets WHERE local_campaign_id = '<campaign_number>';" - All assets recorded: ✅ / ❌
- Command:
- Verify files in Box
- Box folder ID:
__________________________ - All files present: ✅ / ❌
- Filenames have tracking IDs: ✅ / ❌
- Box folder ID:
18. Agency Upload Testing (A2→A3)
When agency uploads first localized file:
- File detected in A2→A3 folder
- Filename:
__________________________ - Tracking ID parsed:
__________________________
- Filename:
- Monitor processing
- Command:
tail -f logs/a2_to_a3.log
- Command:
- Verify steps:
- Downloaded from Box: ✅ / ❌
- Master metadata loaded: ✅ / ❌
- CreativeX score found in database: ✅ / ❌ / N/A
- CreativeX default used if missing: ✅ / N/A
- Uploaded to DAM: ✅ / ❌
- Deleted from Box: ✅ / ❌
- Email sent: ✅ / ❌
- Check email notification
- CreativeX status shown: ✅ / ❌
- Warning if score missing: ✅ / N/A
19. CreativeX Score Integration
- Upload CreativeX PDF to production folder
- PDF uploaded: ✅ / ❌
- Folder ID:
__________________________
- Run scoring script (manual or wait for cron if configured)
- Command:
python scripts/creativex_scoring_storing.py - Score extracted: ✅ / ❌
- Stored in database: ✅ / ❌
- PDF deleted: ✅ / ❌
- Email sent: ✅ / ❌
- Command:
- Verify A2→A3 picks up score
- Upload matching filename to A2→A3
- Score attached to DAM upload: ✅ / ❌
- Email shows score found: ✅ / ❌
POST-CUTOVER MONITORING
20. First 24 Hours - Hourly Checks
Hour 1-2 (Intensive Monitoring):
- Hour 1: Check all workflow logs for errors
- Command:
grep -i error logs/cron_*.log - No critical errors: ✅ / ❌
- Command:
- Hour 2: Verify cron executions
- Command:
grep CRON /var/log/syslog | grep ferrero | tail -20 - All workflows executing: ✅ / ❌
- Command:
Hour 3-6 (Regular Monitoring):
- Hour 3: Check database growth
- Command: See CUTOVER.md monitoring section
- Records increasing: ✅ / ❌
- Hour 4: Check Box folders
- Files being created/deleted: ✅ / ❌
- Hour 5: Verify emails sending
- Emails arriving: ✅ / ❌
- Hour 6: Check for any stuck workflows
- No stuck processes: ✅ / ❌
Hour 7-24 (Light Monitoring):
- Hour 12: Review any errors
- Error count:
__________ - Critical errors: ✅ None / ❌ Found
- Error count:
- Hour 18: Check disk space
- Command:
df -h /opt/ferrero-automation - Space available:
__________ GB
- Command:
- Hour 24: Review day 1 summary
- Campaigns processed:
__________ - Assets processed:
__________ - Success rate:
__________%
- Campaigns processed:
21. Daily Monitoring (First Week)
Day 1
- Review daily report email (7 PM)
- Email received: ✅ / ❌
- Statistics accurate: ✅ / ❌
- Success rate:
__________% - Errors:
__________________________
- Check backup completed
- Command:
./database/check_backups.sh - Backup at 2 AM: ✅ / ❌
- Backup file size:
__________
- Command:
- Review workflow logs
- A1→A2 campaigns:
__________ - A2→A3 uploads:
__________ - A5→A6 rejections:
__________ - Errors:
__________________________
- A1→A2 campaigns:
- Check database records
- Total master assets:
__________ - Total CreativeX scores:
__________ - Total campaigns:
__________
- Total master assets:
Day 2
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Backup health check
- Last backup age:
__________ hours
- Last backup age:
- Log review for errors
- Critical errors:
__________
- Critical errors:
- Stakeholder update sent
- Email sent: ✅ / ❌
Day 3
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Backup verified
- 3 daily backups present: ✅ / ❌
- CreativeX integration working
- Scores being used in A2→A3: ✅ / ❌
Day 4
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Mid-week status update to team
- Update sent: ✅ / ❌
- Check temp directory cleanup
- Command:
du -sh temp/ - Size reasonable: ✅ / ❌
- Command:
Day 5
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Review any recurring errors
- Recurring issues:
__________________________
- Recurring issues:
- Check cron logs for patterns
- All workflows executing: ✅ / ❌
Day 6
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Prepare for weekend monitoring
- On-call person identified:
__________________________
- On-call person identified:
Day 7
- Daily report reviewed
- Success rate:
__________%
- Success rate:
- Verify Sunday weekly backup
- Backup at 3 AM Sunday: ✅ / ❌
- Basebackup file created: ✅ / ❌
- Week 1 summary report
- Total campaigns:
__________ - Total assets:
__________ - Overall success rate:
__________% - Issues resolved:
__________ - Outstanding issues:
__________________________
- Total campaigns:
VALIDATION & SIGN-OFF
22. Week 1 Validation
- All workflows running successfully
- A1→A2: ✅ / ❌
- A5→A6: ✅ / ❌
- B1→B2: ✅ / ❌
- A2→A3: ✅ / ❌
- CreativeX: ✅ / ❌
- Email notifications working
- Success emails: ✅ / ❌
- Error emails: ✅ / ❌
- Daily reports: ✅ / ❌
- Webhooks delivering
- Make.com receiving: ✅ / ❌
- Backups completing daily
- 7 daily backups: ✅ / ❌
- 1 weekly backup: ✅ / ❌
- Database healthy
- No corruption: ✅ / ❌
- Performance good: ✅ / ❌
- No critical errors
- Error rate < 5%: ✅ / ❌
- Stakeholders satisfied
- Ferrero approval: ✅ / ❌
- Oliver approval: ✅ / ❌
23. Production Sign-Off
-
Technical validation complete
- Signed: _________________ Date: _____________
- Name: Dave Porter
- Role: Technical Lead
-
Operations validation complete
- Signed: _________________ Date: _____________
- Name: __________________________
- Role: Operations Manager
-
Business validation complete
- Signed: _________________ Date: _____________
- Name: __________________________
- Role: Ferrero Stakeholder
ROLLBACK PROCEDURES (If Needed)
24. Emergency Rollback
Only if critical production issues found:
-
Stop production cron jobs immediately
- Command:
crontab -e(comment out all lines) - Cron stopped: ✅ / ❌
- Command:
-
Restore staging .env
- Command:
cp .env.staging.backup .env - ENV=staging set: ✅ / ❌
- Command:
-
Restore staging Box-config.json
- Command:
cp ../Box-config.json.staging.backup ../Box-config.json - Staging config restored: ✅ / ❌
- Command:
-
Restore database (if needed)
- Command:
./database/restore.sh <pre_cutover_backup> - Database restored: ✅ / ❌
- Command:
-
Notify stakeholders of rollback
- Email sent: ✅ / ❌
- Reason documented:
__________________________
-
Document what went wrong
- Issue description:
__________________________ - Root cause:
__________________________ - Resolution plan:
__________________________
- Issue description:
-
Resume staging cron jobs
- Staging cron enabled: ✅ / ❌
COMPLETION
25. Final Steps
- Update documentation with production details
- Production URLs documented: ✅ / ❌
- Production folder IDs documented: ✅ / ❌
- Archive staging backups
- Staging backups moved to archive: ✅ / ❌
- Update README.md with production status
- Status updated: ✅ / ❌
- Schedule Week 2 review meeting
- Meeting scheduled: ✅ / ❌
- Date:
__________________________
- Close cutover project
- Cutover complete: ✅ / ❌
- Date:
__________________________ - Final notes:
__________________________
NOTES & ISSUES LOG
Issues Encountered During Cutover
Issue 1:
- Description:
__________________________ - Severity: High / Medium / Low
- Resolution:
__________________________ - Resolved by:
__________________________ - Date/Time:
__________________________
Issue 2:
- Description:
__________________________ - Severity: High / Medium / Low
- Resolution:
__________________________ - Resolved by:
__________________________ - Date/Time:
__________________________
Issue 3:
- Description:
__________________________ - Severity: High / Medium / Low
- Resolution:
__________________________ - Resolved by:
__________________________ - Date/Time:
__________________________
Lessons Learned
What went well:
______________________________________________________________________________
What could be improved:
______________________________________________________________________________
Recommendations for future:
______________________________________________________________________________
QUICK REFERENCE
Important Commands
# Check system status
./database/check_backups.sh
python scripts/test_connection.py
docker ps | grep ferrero
# View logs
tail -f logs/*.log
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"
# Database queries
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"
# Manual workflow execution
python scripts/a1_to_a2_download.py
python scripts/a2_to_a3_upload_polling.py
python scripts/creativex_scoring_storing.py
# Backup and restore
./database/backup.sh --daily
./database/restore.sh --list
Emergency Contacts
- Technical Lead: Dave Porter - daveporter@oliver.agency
- Operations: operations@oliver.agency
- Ferrero IT: [TO BE PROVIDED]
- Escalation: [TO BE PROVIDED]
Critical File Locations
- Configuration:
/opt/ferrero-automation/Python-Version/.env - Box JWT:
/opt/ferrero-automation/Box-config.json - Logs:
/opt/ferrero-automation/Python-Version/logs/ - Backups:
/opt/ferrero-automation/Python-Version/backups/ - Database: Docker container
ferrero-tracking-dbport 5437
Cutover Status: ⬜ Not Started | 🟡 In Progress | ✅ Complete | ❌ Rolled Back
Current Status: __________________________
Cutover Completed: _____________ By: _____________
Production Verified: _____________ By: _____________