841 lines
27 KiB
Markdown
841 lines
27 KiB
Markdown
# Ferrero Automation - Production Cutover TODO List
|
|
|
|
**Status:** Not Started
|
|
**Target Go-Live Date:** _____________
|
|
**Completed By:** _____________
|
|
|
|
---
|
|
|
|
## PRE-CUTOVER PHASE
|
|
|
|
### 1. Obtain Production Credentials
|
|
|
|
#### DAM Production
|
|
- [ ] Confirm production DAM base URL with Ferrero IT
|
|
- Current (staging): `https://ppr.dam.ferrero.com/otmmapi`
|
|
- Production: `__________________________`
|
|
- [ ] Confirm production DAM auth URL
|
|
- Current (staging): `https://ppr.dam.ferrero.com/otdsws/oauth2/token`
|
|
- Production: `__________________________`
|
|
- [ ] Obtain production OAuth2 client ID
|
|
- Production value: `__________________________`
|
|
- [ ] Obtain production OAuth2 client secret
|
|
- Production value: `__________________________`
|
|
- [ ] Test DAM authentication against production endpoint
|
|
- Test command: `python scripts/test_connection.py`
|
|
- Result: ✅ / ❌
|
|
|
|
#### Box Production
|
|
- [ ] Create production Box folder for A1→A2 (Local master downloads)
|
|
- Folder ID: `__________________________`
|
|
- Folder name: `Ferrero DAM - A1→A2 Local Masters`
|
|
- [ ] Create production Box folder for A2→A3 (Agency uploads)
|
|
- Folder ID: `__________________________`
|
|
- Folder name: `Ferrero DAM - A2→A3 Agency Processing`
|
|
- [ ] Create production Box folder for B1→B2 (Global masters)
|
|
- Folder ID: `__________________________`
|
|
- Folder name: `Ferrero DAM - B1→B2 Global Masters`
|
|
- [ ] Create production Box folder for CreativeX PDFs
|
|
- Folder ID: `__________________________`
|
|
- Folder name: `Ferrero DAM - CreativeX Scoring`
|
|
- [ ] Obtain production Box JWT credentials
|
|
- Client ID: `__________________________`
|
|
- Client Secret: `__________________________`
|
|
- JWT Key ID: `__________________________`
|
|
- Enterprise ID: `__________________________`
|
|
- [ ] Generate production Box-config.json file
|
|
- File location: `../Box-config.json`
|
|
- Permissions set: `chmod 600 ../Box-config.json`
|
|
- [ ] Test Box authentication
|
|
- Test result: ✅ / ❌
|
|
|
|
#### Database Production
|
|
- [ ] Decide: Local PostgreSQL or managed database service
|
|
- Decision: `__________________________`
|
|
- [ ] Generate strong production database password
|
|
- Password stored in: `__________________________`
|
|
- [ ] If managed DB: Obtain connection details
|
|
- Host: `__________________________`
|
|
- Port: `__________________________`
|
|
- Username: `__________________________`
|
|
|
|
#### Email Production
|
|
- [ ] Set up production Mailgun/SMTP account (or confirm dev account works)
|
|
- SMTP User: `__________________________`
|
|
- Sender Email: `__________________________`
|
|
- [ ] Define production email recipients
|
|
- Error notifications: `__________________________`
|
|
- Success reports: `__________________________`
|
|
- Daily reports: `__________________________`
|
|
- [ ] Test email delivery to production recipients
|
|
- Test result: ✅ / ❌
|
|
|
|
#### Webhooks Production
|
|
- [ ] Create production Make.com scenario
|
|
- Scenario URL: `__________________________`
|
|
- [ ] Configure webhook authentication (if needed)
|
|
- Auth token: `__________________________`
|
|
- [ ] Test webhook delivery
|
|
- Test result: ✅ / ❌
|
|
|
|
#### CreativeX Production
|
|
- [ ] Confirm LlamaCloud API key for production
|
|
- Use dev key: Yes / No
|
|
- Production key: `__________________________`
|
|
- [ ] Verify agent "Creativex-Extract" exists in production
|
|
- Agent verified: ✅ / ❌
|
|
- [ ] Test CreativeX extraction with production credentials
|
|
- Test result: ✅ / ❌
|
|
|
|
#### mTLS Certificate (Optional)
|
|
- [ ] Obtain production mTLS certificate (.pfx file)
|
|
- Certificate location: `config/certificates/dam-mtls-prod.pfx`
|
|
- [ ] Obtain certificate password
|
|
- Password: `__________________________`
|
|
- [ ] Whitelist production server IP with Ferrero IT
|
|
- Server IP: `__________________________`
|
|
- Whitelisted: ✅ / ❌
|
|
- [ ] Test mTLS authentication
|
|
- Test command: `python scripts/test_connection.py --auth-pfx`
|
|
- Result: ✅ / ❌
|
|
|
|
### 2. Backup Current System
|
|
|
|
- [ ] Backup current .env file
|
|
- Command: `cp .env .env.staging.backup.$(date +%Y%m%d)`
|
|
- Backup location: `__________________________`
|
|
- [ ] Backup current Box-config.json
|
|
- Command: `cp ../Box-config.json ../Box-config.json.staging.backup`
|
|
- Backup location: `__________________________`
|
|
- [ ] Backup current database
|
|
- Command: `./database/backup.sh --daily`
|
|
- Backup file: `__________________________`
|
|
- [ ] Backup current cron jobs
|
|
- Command: `crontab -l > cron_staging_backup_$(date +%Y%m%d).txt`
|
|
- Backup file: `__________________________`
|
|
- [ ] Document current configuration
|
|
- DAM URLs: `__________________________`
|
|
- Box folder IDs: `__________________________`
|
|
- Database settings: `__________________________`
|
|
|
|
### 3. Prepare Production Server
|
|
|
|
- [ ] SSH access to production server confirmed
|
|
- Server hostname: `__________________________`
|
|
- Username: `__________________________`
|
|
- [ ] Git repository cloned to `/opt/ferrero-automation/Python-Version`
|
|
- Path verified: ✅ / ❌
|
|
- [ ] Python 3.10+ installed on server
|
|
- Python version: `__________________________`
|
|
- [ ] Docker installed and running
|
|
- Docker version: `__________________________`
|
|
- [ ] PostgreSQL container running on port 5437
|
|
- Container status: ✅ / ❌
|
|
- [ ] Sufficient disk space available
|
|
- Available space: `__________ GB`
|
|
- Required minimum: 10 GB
|
|
|
|
---
|
|
|
|
## INSTALLATION PHASE
|
|
|
|
### 4. Deploy Code to Production Server
|
|
|
|
- [ ] SSH to production server
|
|
- Command: `ssh user@production-server`
|
|
- [ ] Navigate to deployment directory
|
|
- Command: `cd /opt/ferrero-automation/Python-Version`
|
|
- [ ] Pull latest code from Bitbucket
|
|
- Command: `git pull origin main`
|
|
- Latest commit: `__________________________`
|
|
- [ ] Verify new files present
|
|
- `scripts/creativex_scoring_storing.py`: ✅ / ❌
|
|
- `database/backup.sh`: ✅ / ❌
|
|
- `database/restore.sh`: ✅ / ❌
|
|
- `database/check_backups.sh`: ✅ / ❌
|
|
- `CUTOVER.md`: ✅ / ❌
|
|
|
|
### 5. Install Dependencies
|
|
|
|
- [ ] Create virtual environment (if not exists)
|
|
- Command: `python3 -m venv venv`
|
|
- [ ] Activate virtual environment
|
|
- Command: `source venv/bin/activate`
|
|
- [ ] Install Python dependencies
|
|
- Command: `pip install -r requirements.txt`
|
|
- Or: `sudo venv/bin/pip install -r requirements.txt`
|
|
- [ ] Install llama-cloud-services specifically
|
|
- Command: `sudo venv/bin/pip install llama-cloud-services`
|
|
- [ ] Verify installations
|
|
- Command: `pip list | grep -E "boxsdk|psycopg2|llama-cloud"`
|
|
- boxsdk installed: ✅ / ❌
|
|
- psycopg2-binary installed: ✅ / ❌
|
|
- llama-cloud-services installed: ✅ / ❌
|
|
|
|
### 6. Configure Production Environment
|
|
|
|
- [ ] Update .env file with production credentials
|
|
- Command: `nano .env`
|
|
- [ ] Set environment to production
|
|
- `ENV=production` ✅ / ❌
|
|
- [ ] Update DAM URLs and credentials
|
|
- `DAM_BASE_URL=<production>` ✅ / ❌
|
|
- `DAM_AUTH_URL=<production>` ✅ / ❌
|
|
- `DAM_CLIENT_ID=<production>` ✅ / ❌
|
|
- `DAM_CLIENT_SECRET=<production>` ✅ / ❌
|
|
- [ ] Update Box folder IDs
|
|
- `BOX_ROOT_FOLDER_A1_A2=<production>` ✅ / ❌
|
|
- `BOX_ROOT_FOLDER_A2_A3=<production>` ✅ / ❌
|
|
- `BOX_ROOT_FOLDER_B1_B2=<production>` ✅ / ❌
|
|
- `BOX_ROOT_FOLDER_CREATIVEX=<production>` ✅ / ❌
|
|
- [ ] Update database password
|
|
- `DB_PASSWORD=<strong_production_password>` ✅ / ❌
|
|
- [ ] Update email recipients
|
|
- `ERROR_EMAIL=<production_emails>` ✅ / ❌
|
|
- `REPORT_EMAILS=<production_emails>` ✅ / ❌
|
|
- [ ] Update webhook URL
|
|
- `CAMPAIGN_STATUS_WEBHOOK_URL=<production>` ✅ / ❌
|
|
- [ ] Update CreativeX credentials
|
|
- `LLAMA_CLOUD_API_KEY=<production>` ✅ / ❌
|
|
- [ ] Update mTLS settings (if using)
|
|
- `DAM_MTLS_BASE_URL=<production>` ✅ / ❌
|
|
- `DAM_MTLS_CERT_PATH=config/certificates/dam-mtls-prod.pfx` ✅ / ❌
|
|
- `DAM_MTLS_CERT_PASSWORD=<production>` ✅ / ❌
|
|
- [ ] Save .env file
|
|
- [ ] Set secure permissions on .env
|
|
- Command: `chmod 600 .env`
|
|
|
|
- [ ] Update Box-config.json with production credentials
|
|
- Command: `nano ../Box-config.json`
|
|
- Paste production JWT JSON
|
|
- Save and set permissions: `chmod 600 ../Box-config.json`
|
|
|
|
- [ ] Copy production mTLS certificate (if using)
|
|
- Command: `cp /path/to/cert config/certificates/dam-mtls-prod.pfx`
|
|
- Set permissions: `chmod 600 config/certificates/*.pfx`
|
|
|
|
### 7. Initialize Production Database
|
|
|
|
- [ ] Verify PostgreSQL container running
|
|
- Command: `docker ps | grep ferrero-tracking-db`
|
|
- Container running: ✅ / ❌
|
|
- [ ] Create database schema
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -f database/init.sql`
|
|
- [ ] Verify all tables created
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\dt"`
|
|
- Expected tables:
|
|
- `master_assets` ✅ / ❌
|
|
- `derivative_assets` ✅ / ❌
|
|
- `creativex_scores` ✅ / ❌
|
|
- `campaign_status` ✅ / ❌
|
|
- `asset_events` ✅ / ❌
|
|
- `workflow_state` ✅ / ❌
|
|
- [ ] Verify creativex_scores table structure
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores"`
|
|
- Has 10 columns: ✅ / ❌
|
|
- Has 3 indexes: ✅ / ❌
|
|
|
|
### 8. Make Scripts Executable
|
|
|
|
- [ ] Set executable permissions on workflow scripts
|
|
- Command: `chmod +x scripts/*.py`
|
|
- [ ] Set executable permissions on backup scripts
|
|
- Command: `chmod +x database/*.sh`
|
|
- [ ] Verify permissions
|
|
- Command: `ls -la scripts/*.py database/*.sh`
|
|
|
|
---
|
|
|
|
## TESTING PHASE
|
|
|
|
### 9. Test All Connections
|
|
|
|
- [ ] Test database connection
|
|
- Command: `python scripts/test_connection.py`
|
|
- Result: ✅ / ❌
|
|
- Error (if any): `__________________________`
|
|
|
|
- [ ] Test DAM OAuth2 authentication
|
|
- Command: `python scripts/test_connection.py`
|
|
- Result: ✅ / ❌
|
|
- Error (if any): `__________________________`
|
|
|
|
- [ ] Test DAM mTLS authentication (if using)
|
|
- Command: `python scripts/test_connection.py --auth-pfx`
|
|
- Result: ✅ / ❌
|
|
- Error (if any): `__________________________`
|
|
|
|
- [ ] Test Box authentication
|
|
- Command: `python -c "from shared.config_loader import load_config; from shared.box_client import BoxClient; config = load_config('config/config.yaml'); box = BoxClient(config); print('Box:', box.test_connection())"`
|
|
- Result: ✅ / ❌
|
|
- Error (if any): `__________________________`
|
|
|
|
### 10. Test Email Notifications
|
|
|
|
- [ ] Send test email to production recipients
|
|
- Command: See CUTOVER.md "Step 2.4"
|
|
- Email received by ERROR_EMAIL: ✅ / ❌
|
|
- Email received by REPORT_EMAILS: ✅ / ❌
|
|
- Email formatting correct: ✅ / ❌
|
|
|
|
### 11. Test Webhook Delivery
|
|
|
|
- [ ] Send test webhook to production Make.com
|
|
- Command: See CUTOVER.md "Step 2.5"
|
|
- Webhook received: ✅ / ❌
|
|
- Make.com scenario triggered: ✅ / ❌
|
|
|
|
### 12. Test CreativeX Extraction
|
|
|
|
- [ ] Upload test PDF to production CreativeX Box folder
|
|
- PDF uploaded: ✅ / ❌
|
|
- Folder ID: `__________________________`
|
|
- [ ] Run CreativeX scoring script
|
|
- Command: `python scripts/creativex_scoring_storing.py`
|
|
- Extraction successful: ✅ / ❌
|
|
- Score stored in database: ✅ / ❌
|
|
- PDF deleted from Box: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Verify in database
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT * FROM creativex_scores LIMIT 1;"`
|
|
- Record found: ✅ / ❌
|
|
|
|
### 13. Dry Run Workflows (IMPORTANT: Use Small Test Campaign)
|
|
|
|
#### A1→A2 Test (Master Download)
|
|
- [ ] Identify small test campaign in production DAM with status A1
|
|
- Campaign ID: `__________________________`
|
|
- Campaign name: `__________________________`
|
|
- Number of assets: `__________`
|
|
- [ ] Run A1→A2 script manually
|
|
- Command: `python scripts/a1_to_a2_download.py`
|
|
- Assets downloaded: ✅ / ❌
|
|
- Uploaded to Box: ✅ / ❌
|
|
- Tracking IDs generated: ✅ / ❌
|
|
- Stored in database: ✅ / ❌
|
|
- Status updated A1→A2: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Verify files in production Box folder
|
|
- Files present with tracking IDs: ✅ / ❌
|
|
- [ ] Verify database records
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets ORDER BY created_at DESC LIMIT 5;"`
|
|
- Records found: ✅ / ❌
|
|
|
|
#### A2→A3 Test (Agency Upload)
|
|
- [ ] Upload test file to production A2→A3 Box folder
|
|
- Filename format: `Brand_Country_Language_123_OMG999_<tracking_id>.mp4`
|
|
- Tracking ID matches master asset: ✅ / ❌
|
|
- [ ] Run A2→A3 script manually
|
|
- Command: `python scripts/a2_to_a3_upload_polling.py`
|
|
- File downloaded from Box: ✅ / ❌
|
|
- Master metadata loaded: ✅ / ❌
|
|
- CreativeX score looked up: ✅ / ❌
|
|
- Uploaded to DAM: ✅ / ❌
|
|
- File deleted from Box: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Verify upload in production DAM
|
|
- Asset ID: `__________________________`
|
|
- Metadata correct: ✅ / ❌
|
|
- CreativeX score attached: ✅ / ❌
|
|
- [ ] Check email notification
|
|
- Shows CreativeX status: ✅ / ❌
|
|
- Default values if score missing: ✅ / ❌
|
|
|
|
#### A5→A6 Test (Rejections)
|
|
- [ ] Identify test campaign with rejected assets (status A5)
|
|
- Campaign ID: `__________________________`
|
|
- Has "NOT APPROVED" assets: ✅ / ❌
|
|
- [ ] Run A5→A6 script
|
|
- Command: `python scripts/a5_to_a6_download.py`
|
|
- Only rejected assets downloaded: ✅ / ❌
|
|
- Tracking IDs reused: ✅ / ❌
|
|
- Rejection comments extracted: ✅ / ❌
|
|
- Email shows rejection details: ✅ / ❌
|
|
|
|
#### B1→B2 Test (Global Masters)
|
|
- [ ] Identify test global campaign (status B1)
|
|
- Campaign ID: `__________________________`
|
|
- Campaign type: "Global comm" ✅ / ❌
|
|
- [ ] Run B1→B2 script
|
|
- Command: `python scripts/b1_to_b2_download.py`
|
|
- Downloaded to separate Box folder: ✅ / ❌
|
|
- MASTERS_ prefix used: ✅ / ❌
|
|
- Email sent (no webhook): ✅ / ❌
|
|
|
|
### 14. Configure Backup System
|
|
|
|
- [ ] Make backup scripts executable
|
|
- Command: `chmod +x database/*.sh`
|
|
- [ ] Create backup directories
|
|
- Command: `mkdir -p backups/{dumps,basebackups,safety,manual}`
|
|
- [ ] Set backup directory permissions
|
|
- Command: `chmod 700 backups`
|
|
- [ ] Run test backup
|
|
- Command: `./database/backup.sh --daily`
|
|
- Backup created: ✅ / ❌
|
|
- Backup file: `__________________________`
|
|
- [ ] Verify backup integrity
|
|
- Command: `gunzip -t backups/dumps/<latest>.sql.gz`
|
|
- Valid: ✅ / ❌
|
|
- [ ] Run backup health check
|
|
- Command: `./database/check_backups.sh`
|
|
- Status: ✅ / ❌
|
|
|
|
---
|
|
|
|
## GO/NO-GO DECISION
|
|
|
|
### 15. Pre-Go-Live Validation
|
|
|
|
**Review all tests completed above. GO LIVE only if ALL criteria met:**
|
|
|
|
- [ ] All production credentials obtained and tested
|
|
- [ ] All connection tests passed (DAM, Box, Database, Email, Webhook)
|
|
- [ ] Test campaign processed successfully through A1→A2
|
|
- [ ] Test file uploaded successfully through A2→A3
|
|
- [ ] CreativeX extraction working
|
|
- [ ] Emails delivering to production recipients
|
|
- [ ] Webhook triggering Make.com scenario
|
|
- [ ] Database backups configured and tested
|
|
- [ ] Rollback plan documented and understood
|
|
- [ ] Team trained and ready to monitor
|
|
- [ ] Emergency contacts documented
|
|
|
|
**GO/NO-GO Decision:**
|
|
- [ ] **GO** - Proceed with cutover
|
|
- [ ] **NO-GO** - Delay cutover, address issues above
|
|
|
|
**Decision Made By:** _____________ **Date:** _____________ **Time:** _____________
|
|
|
|
---
|
|
|
|
## GO-LIVE PHASE
|
|
|
|
### 16. Cutover Execution (Go-Live Day)
|
|
|
|
#### Morning Preparation
|
|
|
|
- [ ] Announce cutover to stakeholders
|
|
- Email sent: ✅ / ❌
|
|
- Recipients acknowledged: ✅ / ❌
|
|
- [ ] Stop any running staging workflows
|
|
- Staging cron disabled: ✅ / ❌
|
|
- [ ] Create final pre-cutover backup
|
|
- Command: `./database/backup.sh --daily`
|
|
- Backup file: `__________________________`
|
|
- [ ] Verify production server ready
|
|
- All services running: ✅ / ❌
|
|
- Disk space available: ✅ / ❌
|
|
- Network connectivity: ✅ / ❌
|
|
|
|
#### Enable Production Automation
|
|
|
|
- [ ] Configure production cron jobs
|
|
- Command: `crontab -e`
|
|
- Add all cron jobs from CUTOVER.md
|
|
- [ ] Cron jobs configured:
|
|
- A1→A2 every 5 minutes: ✅ / ❌
|
|
- A5→A6 every 5 minutes: ✅ / ❌
|
|
- B1→B2 every 5 minutes: ✅ / ❌
|
|
- A2→A3 every 5 minutes: ✅ / ❌
|
|
- Daily report at 7 PM: ✅ / ❌
|
|
- Daily backup at 2 AM: ✅ / ❌
|
|
- Weekly backup Sundays 3 AM: ✅ / ❌
|
|
- Backup health check 8 AM: ✅ / ❌
|
|
- [ ] Save crontab
|
|
- [ ] Verify cron service running
|
|
- Command: `systemctl status cron`
|
|
- Status: ✅ / ❌
|
|
|
|
#### Initial Monitoring (First 2 Hours)
|
|
|
|
- [ ] Watch workflow logs
|
|
- Command: `tail -f logs/*.log`
|
|
- Monitoring start time: `__________________________`
|
|
- [ ] Verify first cron execution (wait 5 minutes)
|
|
- A1→A2 executed: ✅ / ❌
|
|
- A5→A6 executed: ✅ / ❌
|
|
- B1→B2 executed: ✅ / ❌
|
|
- A2→A3 executed: ✅ / ❌
|
|
- [ ] Check for errors in first runs
|
|
- Command: `grep -i error logs/cron_*.log`
|
|
- No errors found: ✅ / ❌
|
|
- Errors (if any): `__________________________`
|
|
- [ ] Verify "No campaigns found" is normal
|
|
- Normal if no A1/A5/B1 campaigns ready: ✅ / ❌
|
|
|
|
### 17. First Campaign Processing (Production)
|
|
|
|
**When first production campaign reaches A1 status:**
|
|
|
|
- [ ] Campaign detected in logs
|
|
- Campaign ID: `__________________________`
|
|
- Campaign name: `__________________________`
|
|
- Detected at: `__________________________`
|
|
- [ ] Monitor A1→A2 processing
|
|
- Command: `tail -f logs/a1_to_a2.log`
|
|
- [ ] Verify each step:
|
|
- Assets downloaded from DAM: ✅ / ❌
|
|
- Uploaded to Box with tracking IDs: ✅ / ❌
|
|
- Stored in database: ✅ / ❌
|
|
- Status updated A1→A2: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Check database records
|
|
- Command: `PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT tracking_id, original_filename FROM master_assets WHERE local_campaign_id = '<campaign_number>';"`
|
|
- All assets recorded: ✅ / ❌
|
|
- [ ] Verify files in Box
|
|
- Box folder ID: `__________________________`
|
|
- All files present: ✅ / ❌
|
|
- Filenames have tracking IDs: ✅ / ❌
|
|
|
|
### 18. Agency Upload Testing (A2→A3)
|
|
|
|
**When agency uploads first localized file:**
|
|
|
|
- [ ] File detected in A2→A3 folder
|
|
- Filename: `__________________________`
|
|
- Tracking ID parsed: `__________________________`
|
|
- [ ] Monitor processing
|
|
- Command: `tail -f logs/a2_to_a3.log`
|
|
- [ ] Verify steps:
|
|
- Downloaded from Box: ✅ / ❌
|
|
- Master metadata loaded: ✅ / ❌
|
|
- CreativeX score found in database: ✅ / ❌ / N/A
|
|
- CreativeX default used if missing: ✅ / N/A
|
|
- Uploaded to DAM: ✅ / ❌
|
|
- Deleted from Box: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Check email notification
|
|
- CreativeX status shown: ✅ / ❌
|
|
- Warning if score missing: ✅ / N/A
|
|
|
|
### 19. CreativeX Score Integration
|
|
|
|
- [ ] Upload CreativeX PDF to production folder
|
|
- PDF uploaded: ✅ / ❌
|
|
- Folder ID: `__________________________`
|
|
- [ ] Run scoring script (manual or wait for cron if configured)
|
|
- Command: `python scripts/creativex_scoring_storing.py`
|
|
- Score extracted: ✅ / ❌
|
|
- Stored in database: ✅ / ❌
|
|
- PDF deleted: ✅ / ❌
|
|
- Email sent: ✅ / ❌
|
|
- [ ] Verify A2→A3 picks up score
|
|
- Upload matching filename to A2→A3
|
|
- Score attached to DAM upload: ✅ / ❌
|
|
- Email shows score found: ✅ / ❌
|
|
|
|
---
|
|
|
|
## POST-CUTOVER MONITORING
|
|
|
|
### 20. First 24 Hours - Hourly Checks
|
|
|
|
**Hour 1-2 (Intensive Monitoring):**
|
|
- [ ] Hour 1: Check all workflow logs for errors
|
|
- Command: `grep -i error logs/cron_*.log`
|
|
- No critical errors: ✅ / ❌
|
|
- [ ] Hour 2: Verify cron executions
|
|
- Command: `grep CRON /var/log/syslog | grep ferrero | tail -20`
|
|
- All workflows executing: ✅ / ❌
|
|
|
|
**Hour 3-6 (Regular Monitoring):**
|
|
- [ ] Hour 3: Check database growth
|
|
- Command: See CUTOVER.md monitoring section
|
|
- Records increasing: ✅ / ❌
|
|
- [ ] Hour 4: Check Box folders
|
|
- Files being created/deleted: ✅ / ❌
|
|
- [ ] Hour 5: Verify emails sending
|
|
- Emails arriving: ✅ / ❌
|
|
- [ ] Hour 6: Check for any stuck workflows
|
|
- No stuck processes: ✅ / ❌
|
|
|
|
**Hour 7-24 (Light Monitoring):**
|
|
- [ ] Hour 12: Review any errors
|
|
- Error count: `__________`
|
|
- Critical errors: ✅ None / ❌ Found
|
|
- [ ] Hour 18: Check disk space
|
|
- Command: `df -h /opt/ferrero-automation`
|
|
- Space available: `__________ GB`
|
|
- [ ] Hour 24: Review day 1 summary
|
|
- Campaigns processed: `__________`
|
|
- Assets processed: `__________`
|
|
- Success rate: `__________%`
|
|
|
|
### 21. Daily Monitoring (First Week)
|
|
|
|
#### Day 1
|
|
- [ ] Review daily report email (7 PM)
|
|
- Email received: ✅ / ❌
|
|
- Statistics accurate: ✅ / ❌
|
|
- Success rate: `__________%`
|
|
- Errors: `__________________________`
|
|
- [ ] Check backup completed
|
|
- Command: `./database/check_backups.sh`
|
|
- Backup at 2 AM: ✅ / ❌
|
|
- Backup file size: `__________`
|
|
- [ ] Review workflow logs
|
|
- A1→A2 campaigns: `__________`
|
|
- A2→A3 uploads: `__________`
|
|
- A5→A6 rejections: `__________`
|
|
- Errors: `__________________________`
|
|
- [ ] Check database records
|
|
- Total master assets: `__________`
|
|
- Total CreativeX scores: `__________`
|
|
- Total campaigns: `__________`
|
|
|
|
#### Day 2
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Backup health check
|
|
- Last backup age: `__________ hours`
|
|
- [ ] Log review for errors
|
|
- Critical errors: `__________`
|
|
- [ ] Stakeholder update sent
|
|
- Email sent: ✅ / ❌
|
|
|
|
#### Day 3
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Backup verified
|
|
- 3 daily backups present: ✅ / ❌
|
|
- [ ] CreativeX integration working
|
|
- Scores being used in A2→A3: ✅ / ❌
|
|
|
|
#### Day 4
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Mid-week status update to team
|
|
- Update sent: ✅ / ❌
|
|
- [ ] Check temp directory cleanup
|
|
- Command: `du -sh temp/`
|
|
- Size reasonable: ✅ / ❌
|
|
|
|
#### Day 5
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Review any recurring errors
|
|
- Recurring issues: `__________________________`
|
|
- [ ] Check cron logs for patterns
|
|
- All workflows executing: ✅ / ❌
|
|
|
|
#### Day 6
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Prepare for weekend monitoring
|
|
- On-call person identified: `__________________________`
|
|
|
|
#### Day 7
|
|
- [ ] Daily report reviewed
|
|
- Success rate: `__________%`
|
|
- [ ] Verify Sunday weekly backup
|
|
- Backup at 3 AM Sunday: ✅ / ❌
|
|
- Basebackup file created: ✅ / ❌
|
|
- [ ] Week 1 summary report
|
|
- Total campaigns: `__________`
|
|
- Total assets: `__________`
|
|
- Overall success rate: `__________%`
|
|
- Issues resolved: `__________`
|
|
- Outstanding issues: `__________________________`
|
|
|
|
---
|
|
|
|
## VALIDATION & SIGN-OFF
|
|
|
|
### 22. Week 1 Validation
|
|
|
|
- [ ] All workflows running successfully
|
|
- A1→A2: ✅ / ❌
|
|
- A5→A6: ✅ / ❌
|
|
- B1→B2: ✅ / ❌
|
|
- A2→A3: ✅ / ❌
|
|
- CreativeX: ✅ / ❌
|
|
- [ ] Email notifications working
|
|
- Success emails: ✅ / ❌
|
|
- Error emails: ✅ / ❌
|
|
- Daily reports: ✅ / ❌
|
|
- [ ] Webhooks delivering
|
|
- Make.com receiving: ✅ / ❌
|
|
- [ ] Backups completing daily
|
|
- 7 daily backups: ✅ / ❌
|
|
- 1 weekly backup: ✅ / ❌
|
|
- [ ] Database healthy
|
|
- No corruption: ✅ / ❌
|
|
- Performance good: ✅ / ❌
|
|
- [ ] No critical errors
|
|
- Error rate < 5%: ✅ / ❌
|
|
- [ ] Stakeholders satisfied
|
|
- Ferrero approval: ✅ / ❌
|
|
- Oliver approval: ✅ / ❌
|
|
|
|
### 23. Production Sign-Off
|
|
|
|
- [ ] Technical validation complete
|
|
- **Signed:** _________________ **Date:** _____________
|
|
- **Name:** Dave Porter
|
|
- **Role:** Technical Lead
|
|
|
|
- [ ] Operations validation complete
|
|
- **Signed:** _________________ **Date:** _____________
|
|
- **Name:** __________________________
|
|
- **Role:** Operations Manager
|
|
|
|
- [ ] Business validation complete
|
|
- **Signed:** _________________ **Date:** _____________
|
|
- **Name:** __________________________
|
|
- **Role:** Ferrero Stakeholder
|
|
|
|
---
|
|
|
|
## ROLLBACK PROCEDURES (If Needed)
|
|
|
|
### 24. Emergency Rollback
|
|
|
|
**Only if critical production issues found:**
|
|
|
|
- [ ] Stop production cron jobs immediately
|
|
- Command: `crontab -e` (comment out all lines)
|
|
- Cron stopped: ✅ / ❌
|
|
|
|
- [ ] Restore staging .env
|
|
- Command: `cp .env.staging.backup .env`
|
|
- ENV=staging set: ✅ / ❌
|
|
|
|
- [ ] Restore staging Box-config.json
|
|
- Command: `cp ../Box-config.json.staging.backup ../Box-config.json`
|
|
- Staging config restored: ✅ / ❌
|
|
|
|
- [ ] Restore database (if needed)
|
|
- Command: `./database/restore.sh <pre_cutover_backup>`
|
|
- Database restored: ✅ / ❌
|
|
|
|
- [ ] Notify stakeholders of rollback
|
|
- Email sent: ✅ / ❌
|
|
- Reason documented: `__________________________`
|
|
|
|
- [ ] Document what went wrong
|
|
- Issue description: `__________________________`
|
|
- Root cause: `__________________________`
|
|
- Resolution plan: `__________________________`
|
|
|
|
- [ ] Resume staging cron jobs
|
|
- Staging cron enabled: ✅ / ❌
|
|
|
|
---
|
|
|
|
## COMPLETION
|
|
|
|
### 25. Final Steps
|
|
|
|
- [ ] Update documentation with production details
|
|
- Production URLs documented: ✅ / ❌
|
|
- Production folder IDs documented: ✅ / ❌
|
|
- [ ] Archive staging backups
|
|
- Staging backups moved to archive: ✅ / ❌
|
|
- [ ] Update README.md with production status
|
|
- Status updated: ✅ / ❌
|
|
- [ ] Schedule Week 2 review meeting
|
|
- Meeting scheduled: ✅ / ❌
|
|
- Date: `__________________________`
|
|
- [ ] Close cutover project
|
|
- Cutover complete: ✅ / ❌
|
|
- Date: `__________________________`
|
|
- Final notes: `__________________________`
|
|
|
|
---
|
|
|
|
## NOTES & ISSUES LOG
|
|
|
|
### Issues Encountered During Cutover
|
|
|
|
**Issue 1:**
|
|
- Description: `__________________________`
|
|
- Severity: High / Medium / Low
|
|
- Resolution: `__________________________`
|
|
- Resolved by: `__________________________`
|
|
- Date/Time: `__________________________`
|
|
|
|
**Issue 2:**
|
|
- Description: `__________________________`
|
|
- Severity: High / Medium / Low
|
|
- Resolution: `__________________________`
|
|
- Resolved by: `__________________________`
|
|
- Date/Time: `__________________________`
|
|
|
|
**Issue 3:**
|
|
- Description: `__________________________`
|
|
- Severity: High / Medium / Low
|
|
- Resolution: `__________________________`
|
|
- Resolved by: `__________________________`
|
|
- Date/Time: `__________________________`
|
|
|
|
### Lessons Learned
|
|
|
|
**What went well:**
|
|
1. `__________________________`
|
|
2. `__________________________`
|
|
3. `__________________________`
|
|
|
|
**What could be improved:**
|
|
1. `__________________________`
|
|
2. `__________________________`
|
|
3. `__________________________`
|
|
|
|
**Recommendations for future:**
|
|
1. `__________________________`
|
|
2. `__________________________`
|
|
3. `__________________________`
|
|
|
|
---
|
|
|
|
## QUICK REFERENCE
|
|
|
|
### Important Commands
|
|
|
|
```bash
|
|
# Check system status
|
|
./database/check_backups.sh
|
|
python scripts/test_connection.py
|
|
docker ps | grep ferrero
|
|
|
|
# View logs
|
|
tail -f logs/*.log
|
|
grep -i error logs/*.log | grep "$(date +%Y-%m-%d)"
|
|
|
|
# Database queries
|
|
PGPASSWORD=<prod_password> psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT COUNT(*) FROM master_assets;"
|
|
|
|
# Manual workflow execution
|
|
python scripts/a1_to_a2_download.py
|
|
python scripts/a2_to_a3_upload_polling.py
|
|
python scripts/creativex_scoring_storing.py
|
|
|
|
# Backup and restore
|
|
./database/backup.sh --daily
|
|
./database/restore.sh --list
|
|
```
|
|
|
|
### Emergency Contacts
|
|
|
|
- **Technical Lead:** Dave Porter - daveporter@oliver.agency
|
|
- **Operations:** operations@oliver.agency
|
|
- **Ferrero IT:** [TO BE PROVIDED]
|
|
- **Escalation:** [TO BE PROVIDED]
|
|
|
|
### Critical File Locations
|
|
|
|
- Configuration: `/opt/ferrero-automation/Python-Version/.env`
|
|
- Box JWT: `/opt/ferrero-automation/Box-config.json`
|
|
- Logs: `/opt/ferrero-automation/Python-Version/logs/`
|
|
- Backups: `/opt/ferrero-automation/Python-Version/backups/`
|
|
- Database: Docker container `ferrero-tracking-db` port 5437
|
|
|
|
---
|
|
|
|
**Cutover Status:** ⬜ Not Started | 🟡 In Progress | ✅ Complete | ❌ Rolled Back
|
|
|
|
**Current Status:** __________________________
|
|
|
|
**Cutover Completed:** _____________ **By:** _____________
|
|
|
|
**Production Verified:** _____________ **By:** _____________
|