# Ferrero Content Scaling - Production Server Deployment Guide **Target Server:** Shared hosting with Python 3.6 **Environment:** Production **Requirements:** Python 3.6+, PostgreSQL, SSH access, cron access --- ## 📋 Pre-Deployment Checklist ### Server Requirements - [ ] Python 3.6 or higher installed - [ ] SSH access to server - [ ] Ability to create virtual environments (`python3.6 -m venv`) - [ ] Cron/crontab access - [ ] PostgreSQL access (host/port/credentials) - [ ] Sufficient disk space (500MB minimum) - [ ] Network access to: - ppr.dam.ferrero.com (DAM API) - api.box.com (Box API) - smtp.mailgun.org (Email) - hook.us1.make.celonis.com (Webhook) ### Credentials Needed - [ ] DAM OAuth credentials (client_id, client_secret) - [ ] Box JWT credentials (from Box-config.json) - [ ] PostgreSQL connection details - [ ] Mailgun SMTP credentials - [ ] Make.com webhook URL --- ## 🚀 Step-by-Step Deployment ### Step 1: Upload Files to Server **Option A: Via Git (Recommended)** ```bash # SSH into server ssh user@yourserver.com # Clone repository cd ~ git clone https://bitbucket.org/zlalani/ferrero-opentext.git cd ferrero-opentext/Python-Version ``` **Option B: Via SCP/SFTP** ```bash # From your local machine scp -r Python-Version user@yourserver.com:~/ferrero-automation ``` --- ### Step 2: Verify Python Version ```bash # Check Python version (must be 3.6+) python3 --version # Or specifically check for Python 3.6 python3.6 --version # Verify which Python to use which python3 which python3.6 ``` **Expected output:** ``` Python 3.6.x or higher ``` **If Python 3.6 not found:** - Contact hosting provider to install Python 3.6+ - Or check if it's available at a different path (e.g., `/usr/local/bin/python3.6`) --- ### Step 3: Run Setup Script ```bash cd ~/ferrero-automation # Or wherever you uploaded files # Make setup script executable chmod +x setup.sh # Run setup (will create venv and install dependencies) ./setup.sh ``` **Expected output:** ``` ========================================== Ferrero Content Scaling Automation Setup ========================================== ✓ Found Python 3.6 (server compatible) Step 1: Creating virtual environment... Step 2: Activating virtual environment... Step 3: Upgrading pip... Step 4: Installing dependencies... Step 5: Creating directory structure... Step 6: Creating .env file... ✓ Created .env file - PLEASE EDIT WITH YOUR CREDENTIALS Step 7: Setting file permissions... Step 8: Testing Python imports... ========================================== ✅ Setup Complete! ========================================== ``` **If setup fails:** - Check Python version is 3.6+ - Check you have write permissions - Check internet connection (needs to download packages) --- ### Step 4: Configure Environment Variables ```bash # Edit .env file with your production credentials nano .env ``` **Required Variables:** ```bash # Environment ENV=production # DAM Credentials (from Creds.txt or Postman collection) DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token DAM_CLIENT_ID=otds-OLV DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1 # Box Credentials (from Box-config.json) BOX_CLIENT_ID=l2atwxxq4xna7phcjr2uifm4mbah69qp BOX_CLIENT_SECRET=6XcuCQ6akpk9daE0UHaGSv3mSxWaER4l BOX_JWT_KEY_ID=n1izyn3l BOX_PASSPHRASE=971585f5fd6171428c14a7c8899af5ab BOX_ENTERPRISE_ID=43984435 # Box Folders BOX_ROOT_FOLDER_A1_A2=348304357505 BOX_ROOT_FOLDER_A2_A3=348526703108 # Database (verify with hosting provider) DB_HOST=localhost DB_PORT=5437 DB_USER=ferrero_user DB_PASSWORD=ferrero_pass_2025 # Email (Mailgun SMTP) SMTP_SERVER=smtp.mailgun.org SMTP_PORT=587 SMTP_USER=twist@mail.dev.oliver.solutions SMTP_PASSWORD=102115e9f3b9d7332d0cd1d4329bc0d4-77751bfc-ca066b71 SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency ERROR_EMAIL=daveporter@oliver.agency REPORT_EMAILS=daveporter@oliver.agency # Webhook (Make.com) CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/3f9ztwl8qnljufo0l65utfv5wvvnt9m5 WEBHOOK_AUTH_TOKEN= # Box Webhooks (for A2→A3 - optional for now) BOX_WEBHOOK_PRIMARY_KEY= BOX_WEBHOOK_SECONDARY_KEY= ``` **Important:** - Set `ENV=production` (not staging) - Verify database credentials with hosting provider - Double-check all passwords (no typos!) - Save file: Ctrl+X, Y, Enter --- ### Step 5: Copy Box JWT Config The script needs access to Box-config.json for JWT authentication: ```bash # If Box-config.json is in parent directory (ferrero-opentext/) # Verify the path in config/config.yaml matches # Option A: Verify path is correct (default: ../Box-config.json) cat config/config.yaml | grep rsa_private_key_path # Option B: Or copy Box-config.json to Python-Version directory cp ../Box-config.json ./Box-config.json # Then update config.yaml: nano config/config.yaml # Change: rsa_private_key_path: ./Box-config.json ``` --- ### Step 6: Test Connections ```bash # Activate virtual environment source venv/bin/activate # Test all connections python scripts/test_connection.py ``` **Expected output:** ``` ============================================================ Testing Ferrero Automation Connections ============================================================ ✓ Configuration loaded Testing DAM connection... ✓ DAM connection OK URL: https://ppr.dam.ferrero.com/otmmapi Testing Box connection... ✓ Box connection OK Enterprise ID: 43984435 Testing Database connection... ✓ Database connection OK Host: localhost:5437 ============================================================ Testing complete! ============================================================ ``` **If any connection fails:** - DAM: Check client_id and client_secret (note: 85**o** not 85**0**) - Box: Check Box-config.json path and credentials - Database: Verify host, port, username, password --- ### Step 7: Test A1→A2 Script Manually ```bash # Still in venv python scripts/a1_to_a2_download.py ``` **Expected behavior:** - If A1 campaigns exist: Downloads, uploads to Box, updates status - If no A1 campaigns: "No A1 campaigns found - exiting" **Check logs:** ```bash cat logs/a1_to_a2.log ``` **Verify in Box:** - Go to: https://oliver-na.app.box.com/folder/348304357505 - Look for folder: `C000000078-Campaign_Name` - Files should have tracking IDs in filename --- ### Step 8: Setup Cron Job for A1→A2 **Edit crontab:** ```bash crontab -e ``` **Add this line:** ```bash # Run A1→A2 script every 5 minutes */5 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1 ``` **Important notes:** - Adjust path if you installed in different location - Use full path to python: `~/ferrero-automation/Python-Version/venv/bin/python` - Logs go to `logs/cron.log` **Save and exit:** - Vim: Press `Esc`, type `:wq`, press Enter - Nano: Ctrl+X, Y, Enter **Verify cron is scheduled:** ```bash crontab -l ``` **Monitor cron execution:** ```bash # Watch cron log in real-time tail -f logs/cron.log # Or check last 50 lines tail -50 logs/cron.log ``` --- ### Step 9: Setup Webhook Server for A2→A3 (Optional for now) **Start webhook server in background:** ```bash cd ~/ferrero-automation/Python-Version source venv/bin/activate # Start in background with nohup nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 & # Save process ID echo $! > webhook.pid # Verify it's running ps aux | grep a2_to_a3_upload.py ``` **Create process monitor script:** ```bash nano monitor_webhook.sh ``` **Paste this content:** ```bash #!/bin/bash # Monitor and restart webhook if it stops SCRIPT_DIR="$HOME/ferrero-automation/Python-Version" PID_FILE="$SCRIPT_DIR/webhook.pid" LOG_FILE="$SCRIPT_DIR/logs/monitor.log" # Check if process is running if [ -f "$PID_FILE" ]; then PID=$(cat "$PID_FILE") if ps -p $PID > /dev/null 2>&1; then echo "$(date): Webhook running (PID: $PID)" >> "$LOG_FILE" exit 0 fi fi # Process not running - restart echo "$(date): Webhook not running, restarting..." >> "$LOG_FILE" cd "$SCRIPT_DIR" source venv/bin/activate nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 & echo $! > "$PID_FILE" echo "$(date): Webhook restarted (PID: $(cat $PID_FILE))" >> "$LOG_FILE" ``` **Make executable:** ```bash chmod +x monitor_webhook.sh ``` **Add to crontab (check every 5 minutes):** ```bash crontab -e # Add: */5 * * * * ~/ferrero-automation/Python-Version/monitor_webhook.sh ``` **Note:** A2→A3 webhook can be deployed later when ready to receive Box webhooks --- ## 📊 Verification & Monitoring ### Check Script is Running **View recent cron executions:** ```bash tail -50 logs/cron.log ``` **Check for errors:** ```bash grep -i error logs/a1_to_a2.log | tail -20 ``` **Check successful uploads:** ```bash grep "Campaign completed successfully" logs/a1_to_a2.log ``` ### Monitor Database ```bash # Check recent tracking IDs PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " SELECT tracking_id, original_filename, created_at FROM master_assets ORDER BY created_at DESC LIMIT 10; " # Count assets by status PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " SELECT status, COUNT(*) FROM master_assets GROUP BY status; " ``` ### Check Email Notifications - Check inbox: daveporter@oliver.agency - Success emails: Campaign completed - Error emails: Upload failed or partial completion ### Check Box Uploads - Go to: https://oliver-na.app.box.com/folder/348304357505 - Look for folders: `C000000078-Campaign_Name` - Files should have tracking IDs: `filename_ABC123.mp4` ### Check Webhook - Check Make.com scenario for incoming data - Payload includes: campaign_id, campaign_number, asset_count, timestamp --- ## 🔧 Troubleshooting ### Script Not Running **Check cron is scheduled:** ```bash crontab -l ``` **Check cron log for errors:** ```bash tail -50 logs/cron.log ``` **Run manually to see errors:** ```bash cd ~/ferrero-automation/Python-Version source venv/bin/activate python scripts/a1_to_a2_download.py ``` ### No A1 Campaigns Found **Verify campaigns exist:** ```bash python -c " import sys sys.path.insert(0, 'scripts') from shared.config_loader import load_config from shared.dam_client import DAMClient config = load_config('config/config.yaml') dam = DAMClient(config) # Search without status filter campaigns = dam.search_campaigns() print('Total campaigns:', len(campaigns)) # Search for A1 a1_campaigns = dam.search_campaigns(status='A1') print('A1 campaigns:', len(a1_campaigns)) " ``` ### Connection Errors **DAM Connection Failed:** - Check client_secret has lowercase 'o' not zero: `85oO` not `850` - Verify URL: https://ppr.dam.ferrero.com/otmmapi - Test OAuth manually in PHP app to verify credentials still valid **Box Connection Failed:** - Verify Box-config.json path is correct - Check JWT credentials haven't expired - Verify enterprise ID: 43984435 **Database Connection Failed:** - Check host and port with hosting provider - Verify username and password - Test connection: `psql -h HOST -p PORT -U USER -d ferrero_tracking` ### Emails Not Sending **Check SMTP credentials:** ```bash # Test SMTP connection python -c " import smtplib server = smtplib.SMTP('smtp.mailgun.org', 587) server.starttls() server.login('twist@mail.dev.oliver.solutions', '102115e9f3b9d7332d0cd1d4329bc0d4-77751bfc-ca066b71') print('SMTP connection OK') server.quit() " ``` **Check email template syntax:** - HTML templates use `{{ variable }}` (Jinja2) - Subject uses `{variable}` (Python format) ### Box Folder Not Created **Check root folder ID:** ```bash grep BOX_ROOT_FOLDER .env # Should show: BOX_ROOT_FOLDER_A1_A2=348304357505 ``` **Check folder permissions:** - Verify service account has write access to folder 348304357505 --- ## 🔐 Security Best Practices ### Protect Credentials ```bash # Secure .env file (only owner can read) chmod 600 .env # Verify permissions ls -la .env # Should show: -rw------- (600) # Never commit .env to git echo ".env" >> .gitignore ``` ### Log File Security ```bash # Restrict log file access chmod 640 logs/*.log # Rotate logs periodically (optional) # Add to crontab: 0 0 * * 0 cd ~/ferrero-automation/Python-Version && find logs/ -name "*.log" -mtime +30 -delete ``` --- ## 📝 Configuration for Production ### Update config/config.yaml **Create production environment config:** ```bash nano config/environments/production.yaml ``` **Add production-specific settings:** ```yaml # Production overrides # Polling - more frequent in production polling: interval_seconds: 180 # 3 minutes instead of 5 # Logging - less verbose logging: level: INFO # Not DEBUG # Notifications - production recipients notifications: recipients: success: - team@ferrero.com - agency@example.com errors: - admin@ferrero.com - daveporter@oliver.agency critical: - oncall@ferrero.com ``` --- ## 🕐 Cron Job Examples ### A1→A2 Every 5 Minutes ```bash */5 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1 ``` ### A1→A2 Every 3 Minutes (Production) ```bash */3 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1 ``` ### Log Cleanup (Weekly, Sunday at midnight) ```bash 0 0 * * 0 cd ~/ferrero-automation/Python-Version && find logs/ -name "*.log" -mtime +30 -delete ``` ### Temp File Cleanup (Daily at 2 AM) ```bash 0 2 * * * cd ~/ferrero-automation/Python-Version && find temp/downloads/ -mtime +1 -delete ``` --- ## 🔄 Updates and Maintenance ### Update Code from Git ```bash cd ~/ferrero-automation/Python-Version # Pull latest changes git pull # Restart virtual environment if dependencies changed source venv/bin/activate pip install -r requirements.txt --upgrade # Restart webhook server if running if [ -f webhook.pid ]; then kill $(cat webhook.pid) nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 & echo $! > webhook.pid fi ``` ### Add New MVP Field **Edit config/field_mappings.yaml:** ```bash nano config/field_mappings.yaml # Add new field ID to mvp_fields list: mvp_fields: - FERRERO.FIELD.MKTG.ASSET TYPE # ... existing fields ... - NEW.FIELD.ID.HERE # Just add here! # Save and exit (Ctrl+X, Y, Enter) ``` **No code changes needed!** Next script run will use new field. ### Change Polling Interval ```bash # Edit crontab crontab -e # Change from */5 to */3 for 3-minute interval */3 * * * * cd ~/ferrero-automation/Python-Version && ... ``` ### Change Email Recipients ```bash # Edit .env nano .env # Update: ERROR_EMAIL=new-admin@ferrero.com REPORT_EMAILS=team@ferrero.com,manager@ferrero.com # Save and restart ``` --- ## 🐛 Debugging ### Enable Debug Logging ```bash # Temporarily enable debug logging nano config/config.yaml # Change: logging: level: DEBUG # Instead of INFO # Run script manually source venv/bin/activate python scripts/a1_to_a2_download.py # Check detailed logs tail -100 logs/a1_to_a2.log ``` ### Test Individual Components **Test DAM only:** ```bash python -c " import sys sys.path.insert(0, 'scripts') from shared.config_loader import load_config from shared.dam_client import DAMClient config = load_config('config/config.yaml') dam = DAMClient(config) campaigns = dam.search_campaigns(status='A1') print('Found {} A1 campaigns'.format(len(campaigns))) for c in campaigns: print(' -', c.get('campaign_name'), c.get('campaign_id')) " ``` **Test Box only:** ```bash python -c " import sys sys.path.insert(0, 'scripts') from shared.config_loader import load_config from shared.box_client import BoxClient config = load_config('config/config.yaml') box = BoxClient(config) print('Box connection:', box.test_connection()) " ``` **Test Database only:** ```bash python -c " import sys sys.path.insert(0, 'scripts') from shared.config_loader import load_config from shared.database import Database config = load_config('config/config.yaml') db = Database(config) print('Database connection:', db.test_connection()) db.close() " ``` --- ## 📞 Support & Escalation ### Check Script Health **Quick health check script:** ```bash #!/bin/bash # health_check.sh echo "=== Ferrero Automation Health Check ===" echo "" # Check cron is scheduled echo "Cron Jobs:" crontab -l | grep ferrero echo "" echo "Recent Executions:" tail -5 logs/cron.log echo "" echo "Recent Successes:" grep "Campaign completed successfully" logs/a1_to_a2.log | tail -3 echo "" echo "Recent Errors:" grep -i error logs/a1_to_a2.log | tail -5 echo "" echo "Database Records (last 24h):" PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c " SELECT COUNT(*) as count, DATE(created_at) as date FROM master_assets WHERE created_at > NOW() - INTERVAL '24 hours' GROUP BY DATE(created_at); " ``` **Make executable and run:** ```bash chmod +x health_check.sh ./health_check.sh ``` ### If Script Stops Working 1. **Check cron is running:** ```bash service cron status # or: systemctl status cron ``` 2. **Check disk space:** ```bash df -h ``` 3. **Check logs for errors:** ```bash tail -100 logs/a1_to_a2.log | grep -i error ``` 4. **Test manually:** ```bash source venv/bin/activate python scripts/a1_to_a2_download.py ``` 5. **Check database connectivity:** ```bash python scripts/test_connection.py ``` ### Contact Information **For script issues:** - Check logs first: `logs/a1_to_a2.log` - Check cron log: `logs/cron.log` - Email: daveporter@oliver.agency **For server issues:** - Contact hosting provider - Check Python 3.6 is still available - Check cron service is running --- ## 🎯 Production Deployment Checklist ### Pre-Deployment - [ ] Uploaded all files to server - [ ] Created virtual environment - [ ] Installed dependencies (requirements.txt) - [ ] Configured .env with production credentials - [ ] Copied Box-config.json to correct location - [ ] Verified all paths in config.yaml ### Testing - [ ] Ran test_connection.py - all green - [ ] Ran a1_to_a2_download.py manually - worked - [ ] Verified files uploaded to Box (correct folder) - [ ] Verified database records created - [ ] Received email notification - [ ] Webhook sent to Make.com ### Deployment - [ ] Setup cron job for A1→A2 - [ ] Verified cron is scheduled (crontab -l) - [ ] Waited 5 minutes, checked cron.log - [ ] Verified script executed automatically - [ ] Setup A2→A3 webhook server (if ready) - [ ] Setup webhook monitor cron ### Post-Deployment - [ ] Monitor logs for 24 hours - [ ] Check emails are being received - [ ] Verify campaigns progressing through workflow - [ ] Check database growth - [ ] Verify Box folders being created correctly --- ## 📖 Quick Reference Commands ### Daily Operations ```bash # Check if script is working tail -20 logs/a1_to_a2.log # Count processed assets today grep "$(date +%Y-%m-%d)" logs/a1_to_a2.log | grep "Campaign completed" | wc -l # View latest email sent grep "Email sent" logs/a1_to_a2.log | tail -1 # Check webhook status grep "Webhook sent" logs/a1_to_a2.log | tail -5 ``` ### Emergency Procedures **Stop A1→A2 automation:** ```bash # Remove from crontab crontab -e # Comment out or delete the line with a1_to_a2_download.py ``` **Stop A2→A3 webhook:** ```bash # If running kill $(cat webhook.pid) rm webhook.pid ``` **Restart everything:** ```bash # Re-add to crontab crontab -e # Restart webhook cd ~/ferrero-automation/Python-Version source venv/bin/activate nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 & echo $! > webhook.pid ``` --- ## ✅ Success Criteria **A1→A2 automation is working if:** - ✅ Cron runs every 5 minutes (check cron.log) - ✅ A1 campaigns are processed (check a1_to_a2.log) - ✅ Files appear in Box folder 348304357505 - ✅ Database records increase - ✅ Campaign status changes from A1 to A2 - ✅ Emails arrive at daveporter@oliver.agency - ✅ Webhooks arrive at Make.com --- ## 📞 Support **Documentation:** - README.md - Quick start guide - DEPLOYMENT.md - This file - PYTHON_AUTOMATION_PLAN.md - Complete architecture - PROJECT_STATUS_2025-10-30.md - Full session summary **Logs Location:** - Application logs: `logs/a1_to_a2.log` - Cron logs: `logs/cron.log` - Webhook logs: `logs/webhook.log` - Error logs: `logs/errors.log` **Configuration Files:** - Main config: `config/config.yaml` - Field mappings: `config/field_mappings.yaml` - Environment: `.env` - Production overrides: `config/environments/production.yaml` --- **Deployment Version:** 1.0.0 **Python Version Required:** 3.6+ **Tested With:** Python 3.6 (server), Python 3.10 (local) **Status:** Production Ready --- **For additional support, refer to PROJECT_STATUS_2025-10-30.md** --- ## 🎯 Key Features ### A1→A2 Script Features: - ✅ Single-run mode (process one campaign and exit) - ✅ Downloads all master assets from campaign - ✅ Uploads to Box with tracking IDs - ✅ Folder naming: C000000078-Campaign_Name - ✅ All-done check before status update - ✅ Status update A1 → A2 only when all successful - ✅ Webhook notification to Make.com - ✅ **Detailed email** with list of all processed assets (names, tracking IDs, Box URLs) - ✅ **Log rotation** (10MB files, keeps 28 backups = 1 month) - ✅ Retry failed campaigns on next run ### A2→A3 Script Features: - ✅ Polling mode (checks Box folder, no webhook needed) - ✅ Single-run mode (process one file and exit) - ✅ Parses V2 filenames with strict validation - ✅ Loads master metadata from PostgreSQL - ✅ Builds 27 MVP fields from master - ✅ Updates Description, State, Language from filename - ✅ Uploads to DAM with clean filename - ✅ **Deletes file from Box after successful upload** - ✅ **Email notification for each upload** (asset ID, tracking ID, filename) - ✅ **Log rotation** (10MB files, keeps 28 backups = 1 month) - ✅ Processes files one at a time ### Email Notifications: - ✅ **Detailed asset lists** in emails (not just counts) - ✅ Successful uploads: Show asset name, tracking ID, Box URL - ✅ Failed uploads: Show asset name and specific error - ✅ Partial completion: Lists both successful and failed assets - ✅ Per-file notifications for A2→A3 uploads - ✅ Sent via SMTP (Mailgun) - ✅ HTML formatted ### Log Management: - ✅ **Automatic rotation** at 10MB per file - ✅ **Keeps 28 backups** (approximately 1 month) - ✅ Total log space: ~280MB maximum - ✅ Automatic cleanup (old logs auto-deleted) - ✅ No manual maintenance required ### Box File Management: - ✅ **A1→A2:** Uploads to folder 348304357505 - ✅ **A2→A3:** Polls folder 348526703108 - ✅ **Automatic deletion** from Box after DAM upload - ✅ Prevents reprocessing same files - ✅ Keeps Box folders clean