ferrero-opentext/Python-Version/DEPLOYMENT.md

# Ferrero Content Scaling - Production Server Deployment Guide

**Target Server:** Shared hosting with Python 3.6
**Environment:** Production
**Requirements:** Python 3.6+, PostgreSQL, SSH access, cron access

---

## 📋 Pre-Deployment Checklist

### Server Requirements
- [ ] Python 3.6 or higher installed
- [ ] SSH access to server
- [ ] Ability to create virtual environments (`python3.6 -m venv`)
- [ ] Cron/crontab access
- [ ] PostgreSQL access (host/port/credentials)
- [ ] Sufficient disk space (500MB minimum)
- [ ] Network access to:
  - ppr.dam.ferrero.com (DAM API)
  - api.box.com (Box API)
  - smtp.mailgun.org (Email)
  - hook.us1.make.celonis.com (Webhook)

### Credentials Needed
- [ ] DAM OAuth credentials (client_id, client_secret)
- [ ] Box JWT credentials (from Box-config.json)
- [ ] PostgreSQL connection details
- [ ] Mailgun SMTP credentials
- [ ] Make.com webhook URL

---

## 🚀 Step-by-Step Deployment

### Step 1: Upload Files to Server

**Option A: Via Git (Recommended)**
```bash
# SSH into server
ssh user@yourserver.com

# Clone repository
cd ~
git clone https://bitbucket.org/zlalani/ferrero-opentext.git
cd ferrero-opentext/Python-Version
```

**Option B: Via SCP/SFTP**
```bash
# From your local machine
scp -r Python-Version user@yourserver.com:~/ferrero-automation
```

---

### Step 2: Verify Python Version

```bash
# Check Python version (must be 3.6+)
python3 --version

# Or specifically check for Python 3.6
python3.6 --version

# Verify which Python to use
which python3
which python3.6
```

**Expected output:**
```
Python 3.6.x or higher
```

**If Python 3.6 not found:**
- Contact hosting provider to install Python 3.6+
- Or check if it's available at a different path (e.g., `/usr/local/bin/python3.6`)

---

### Step 3: Run Setup Script

```bash
cd ~/ferrero-automation  # Or wherever you uploaded files

# Make setup script executable
chmod +x setup.sh

# Run setup (will create venv and install dependencies)
./setup.sh
```

**Expected output:**
```
==========================================
Ferrero Content Scaling Automation Setup
==========================================

✓ Found Python 3.6 (server compatible)

Step 1: Creating virtual environment...
Step 2: Activating virtual environment...
Step 3: Upgrading pip...
Step 4: Installing dependencies...
Step 5: Creating directory structure...
Step 6: Creating .env file...
✓ Created .env file - PLEASE EDIT WITH YOUR CREDENTIALS
Step 7: Setting file permissions...
Step 8: Testing Python imports...

==========================================
✅ Setup Complete!
==========================================
```

**If setup fails:**
- Check Python version is 3.6+
- Check you have write permissions
- Check internet connection (needs to download packages)

---

### Step 4: Configure Environment Variables

```bash
# Edit .env file with your production credentials
nano .env
```

**Required Variables:**
```bash
# Environment
ENV=production

# DAM Credentials (from Creds.txt or Postman collection)
DAM_BASE_URL=https://ppr.dam.ferrero.com/otmmapi
DAM_AUTH_URL=https://ppr.dam.ferrero.com/otdsws/oauth2/token
DAM_CLIENT_ID=otds-OLV
DAM_CLIENT_SECRET=hs28LZ9ZzQ5I9rlW3P7Wwyw85oOatlC1

# Box Credentials (from Box-config.json)
BOX_CLIENT_ID=l2atwxxq4xna7phcjr2uifm4mbah69qp
BOX_CLIENT_SECRET=6XcuCQ6akpk9daE0UHaGSv3mSxWaER4l
BOX_JWT_KEY_ID=n1izyn3l
BOX_PASSPHRASE=971585f5fd6171428c14a7c8899af5ab
BOX_ENTERPRISE_ID=43984435

# Box Folders
BOX_ROOT_FOLDER_A1_A2=348304357505
BOX_ROOT_FOLDER_A2_A3=348526703108

# Database (verify with hosting provider)
DB_HOST=localhost
DB_PORT=5433
DB_USER=ferrero_user
DB_PASSWORD=ferrero_pass_2025

# Email (Mailgun SMTP)
SMTP_SERVER=smtp.mailgun.org
SMTP_PORT=587
SMTP_USER=twist@mail.dev.oliver.solutions
SMTP_PASSWORD=102115e9f3b9d7332d0cd1d4329bc0d4-77751bfc-ca066b71
SENDER_EMAIL=TWIST-UK-SERVER@oliver.agency
ERROR_EMAIL=daveporter@oliver.agency
REPORT_EMAILS=daveporter@oliver.agency

# Webhook (Make.com)
CAMPAIGN_STATUS_WEBHOOK_URL=https://hook.us1.make.celonis.com/3f9ztwl8qnljufo0l65utfv5wvvnt9m5
WEBHOOK_AUTH_TOKEN=

# Box Webhooks (for A2→A3 - optional for now)
BOX_WEBHOOK_PRIMARY_KEY=
BOX_WEBHOOK_SECONDARY_KEY=
```

**Important:**
- Set `ENV=production` (not staging)
- Verify database credentials with hosting provider
- Double-check all passwords (no typos!)
- Save file: Ctrl+X, Y, Enter

---

### Step 5: Copy Box JWT Config

The script needs access to Box-config.json for JWT authentication:

```bash
# If Box-config.json is in parent directory (ferrero-opentext/)
# Verify the path in config/config.yaml matches

# Option A: Verify path is correct (default: ../Box-config.json)
cat config/config.yaml | grep rsa_private_key_path

# Option B: Or copy Box-config.json to Python-Version directory
cp ../Box-config.json ./Box-config.json

# Then update config.yaml:
nano config/config.yaml
# Change: rsa_private_key_path: ./Box-config.json
```

---

### Step 6: Test Connections

```bash
# Activate virtual environment
source venv/bin/activate

# Test all connections
python scripts/test_connection.py
```

**Expected output:**
```
============================================================
Testing Ferrero Automation Connections
============================================================

✓ Configuration loaded

Testing DAM connection...
✓ DAM connection OK
  URL: https://ppr.dam.ferrero.com/otmmapi

Testing Box connection...
✓ Box connection OK
  Enterprise ID: 43984435

Testing Database connection...
✓ Database connection OK
  Host: localhost:5433

============================================================
Testing complete!
============================================================
```

**If any connection fails:**
- DAM: Check client_id and client_secret (note: 85**o** not 85**0**)
- Box: Check Box-config.json path and credentials
- Database: Verify host, port, username, password

---

### Step 7: Test A1→A2 Script Manually

```bash
# Still in venv
python scripts/a1_to_a2_download.py
```

**Expected behavior:**
- If A1 campaigns exist: Downloads, uploads to Box, updates status
- If no A1 campaigns: "No A1 campaigns found - exiting"

**Check logs:**
```bash
cat logs/a1_to_a2.log
```

**Verify in Box:**
- Go to: https://oliver-na.app.box.com/folder/348304357505
- Look for folder: `C000000078-Campaign_Name`
- Files should have tracking IDs in filename

---

### Step 8: Setup Cron Job for A1→A2

**Edit crontab:**
```bash
crontab -e
```

**Add this line:**
```bash
# Run A1→A2 script every 5 minutes
*/5 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1
```

**Important notes:**
- Adjust path if you installed in different location
- Use full path to python: `~/ferrero-automation/Python-Version/venv/bin/python`
- Logs go to `logs/cron.log`

**Save and exit:**
- Vim: Press `Esc`, type `:wq`, press Enter
- Nano: Ctrl+X, Y, Enter

**Verify cron is scheduled:**
```bash
crontab -l
```

**Monitor cron execution:**
```bash
# Watch cron log in real-time
tail -f logs/cron.log

# Or check last 50 lines
tail -50 logs/cron.log
```

---

### Step 9: Setup Webhook Server for A2→A3 (Optional for now)

**Start webhook server in background:**
```bash
cd ~/ferrero-automation/Python-Version
source venv/bin/activate

# Start in background with nohup
nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 &

# Save process ID
echo $! > webhook.pid

# Verify it's running
ps aux | grep a2_to_a3_upload.py
```

**Create process monitor script:**
```bash
nano monitor_webhook.sh
```

**Paste this content:**
```bash
#!/bin/bash
# Monitor and restart webhook if it stops

SCRIPT_DIR="$HOME/ferrero-automation/Python-Version"
PID_FILE="$SCRIPT_DIR/webhook.pid"
LOG_FILE="$SCRIPT_DIR/logs/monitor.log"

# Check if process is running
if [ -f "$PID_FILE" ]; then
    PID=$(cat "$PID_FILE")
    if ps -p $PID > /dev/null 2>&1; then
        echo "$(date): Webhook running (PID: $PID)" >> "$LOG_FILE"
        exit 0
    fi
fi

# Process not running - restart
echo "$(date): Webhook not running, restarting..." >> "$LOG_FILE"
cd "$SCRIPT_DIR"
source venv/bin/activate
nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 &
echo $! > "$PID_FILE"
echo "$(date): Webhook restarted (PID: $(cat $PID_FILE))" >> "$LOG_FILE"
```

**Make executable:**
```bash
chmod +x monitor_webhook.sh
```

**Add to crontab (check every 5 minutes):**
```bash
crontab -e

# Add:
*/5 * * * * ~/ferrero-automation/Python-Version/monitor_webhook.sh
```

**Note:** A2→A3 webhook can be deployed later when ready to receive Box webhooks

---

## 📊 Verification & Monitoring

### Check Script is Running

**View recent cron executions:**
```bash
tail -50 logs/cron.log
```

**Check for errors:**
```bash
grep -i error logs/a1_to_a2.log | tail -20
```

**Check successful uploads:**
```bash
grep "Campaign completed successfully" logs/a1_to_a2.log
```

### Monitor Database

```bash
# Check recent tracking IDs
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5433 -U ferrero_user -d ferrero_tracking -c "
SELECT tracking_id, original_filename, created_at
FROM master_assets
ORDER BY created_at DESC
LIMIT 10;
"

# Count assets by status
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5433 -U ferrero_user -d ferrero_tracking -c "
SELECT status, COUNT(*)
FROM master_assets
GROUP BY status;
"
```

### Check Email Notifications

- Check inbox: daveporter@oliver.agency
- Success emails: Campaign completed
- Error emails: Upload failed or partial completion

### Check Box Uploads

- Go to: https://oliver-na.app.box.com/folder/348304357505
- Look for folders: `C000000078-Campaign_Name`
- Files should have tracking IDs: `filename_ABC123.mp4`

### Check Webhook

- Check Make.com scenario for incoming data
- Payload includes: campaign_id, campaign_number, asset_count, timestamp

---

## 🔧 Troubleshooting

### Script Not Running

**Check cron is scheduled:**
```bash
crontab -l
```

**Check cron log for errors:**
```bash
tail -50 logs/cron.log
```

**Run manually to see errors:**
```bash
cd ~/ferrero-automation/Python-Version
source venv/bin/activate
python scripts/a1_to_a2_download.py
```

### No A1 Campaigns Found

**Verify campaigns exist:**
```bash
python -c "
import sys
sys.path.insert(0, 'scripts')
from shared.config_loader import load_config
from shared.dam_client import DAMClient

config = load_config('config/config.yaml')
dam = DAMClient(config)

# Search without status filter
campaigns = dam.search_campaigns()
print('Total campaigns:', len(campaigns))

# Search for A1
a1_campaigns = dam.search_campaigns(status='A1')
print('A1 campaigns:', len(a1_campaigns))
"
```

### Connection Errors

**DAM Connection Failed:**
- Check client_secret has lowercase 'o' not zero: `85oO` not `850`
- Verify URL: https://ppr.dam.ferrero.com/otmmapi
- Test OAuth manually in PHP app to verify credentials still valid

**Box Connection Failed:**
- Verify Box-config.json path is correct
- Check JWT credentials haven't expired
- Verify enterprise ID: 43984435

**Database Connection Failed:**
- Check host and port with hosting provider
- Verify username and password
- Test connection: `psql -h HOST -p PORT -U USER -d ferrero_tracking`

### Emails Not Sending

**Check SMTP credentials:**
```bash
# Test SMTP connection
python -c "
import smtplib
server = smtplib.SMTP('smtp.mailgun.org', 587)
server.starttls()
server.login('twist@mail.dev.oliver.solutions', '102115e9f3b9d7332d0cd1d4329bc0d4-77751bfc-ca066b71')
print('SMTP connection OK')
server.quit()
"
```

**Check email template syntax:**
- HTML templates use `{{ variable }}` (Jinja2)
- Subject uses `{variable}` (Python format)

### Box Folder Not Created

**Check root folder ID:**
```bash
grep BOX_ROOT_FOLDER .env
# Should show: BOX_ROOT_FOLDER_A1_A2=348304357505
```

**Check folder permissions:**
- Verify service account has write access to folder 348304357505

---

## 🔐 Security Best Practices

### Protect Credentials

```bash
# Secure .env file (only owner can read)
chmod 600 .env

# Verify permissions
ls -la .env
# Should show: -rw------- (600)

# Never commit .env to git
echo ".env" >> .gitignore
```

### Log File Security

```bash
# Restrict log file access
chmod 640 logs/*.log

# Rotate logs periodically (optional)
# Add to crontab:
0 0 * * 0 cd ~/ferrero-automation/Python-Version && find logs/ -name "*.log" -mtime +30 -delete
```

---

## 📝 Configuration for Production

### Update config/config.yaml

**Create production environment config:**
```bash
nano config/environments/production.yaml
```

**Add production-specific settings:**
```yaml
# Production overrides

# Polling - more frequent in production
polling:
  interval_seconds: 180  # 3 minutes instead of 5

# Logging - less verbose
logging:
  level: INFO  # Not DEBUG

# Notifications - production recipients
notifications:
  recipients:
    success:
      - team@ferrero.com
      - agency@example.com
    errors:
      - admin@ferrero.com
      - daveporter@oliver.agency
    critical:
      - oncall@ferrero.com
```

---

## 🕐 Cron Job Examples

### A1→A2 Every 5 Minutes
```bash
*/5 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1
```

### A1→A2 Every 3 Minutes (Production)
```bash
*/3 * * * * cd ~/ferrero-automation/Python-Version && venv/bin/python scripts/a1_to_a2_download.py >> logs/cron.log 2>&1
```

### Log Cleanup (Weekly, Sunday at midnight)
```bash
0 0 * * 0 cd ~/ferrero-automation/Python-Version && find logs/ -name "*.log" -mtime +30 -delete
```

### Temp File Cleanup (Daily at 2 AM)
```bash
0 2 * * * cd ~/ferrero-automation/Python-Version && find temp/downloads/ -mtime +1 -delete
```

---

## 🔄 Updates and Maintenance

### Update Code from Git

```bash
cd ~/ferrero-automation/Python-Version

# Pull latest changes
git pull

# Restart virtual environment if dependencies changed
source venv/bin/activate
pip install -r requirements.txt --upgrade

# Restart webhook server if running
if [ -f webhook.pid ]; then
    kill $(cat webhook.pid)
    nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 &
    echo $! > webhook.pid
fi
```

### Add New MVP Field

**Edit config/field_mappings.yaml:**
```bash
nano config/field_mappings.yaml

# Add new field ID to mvp_fields list:
mvp_fields:
  - FERRERO.FIELD.MKTG.ASSET TYPE
  # ... existing fields ...
  - NEW.FIELD.ID.HERE  # Just add here!

# Save and exit (Ctrl+X, Y, Enter)
```

**No code changes needed!** Next script run will use new field.

### Change Polling Interval

```bash
# Edit crontab
crontab -e

# Change from */5 to */3 for 3-minute interval
*/3 * * * * cd ~/ferrero-automation/Python-Version && ...
```

### Change Email Recipients

```bash
# Edit .env
nano .env

# Update:
ERROR_EMAIL=new-admin@ferrero.com
REPORT_EMAILS=team@ferrero.com,manager@ferrero.com

# Save and restart
```

---

## 🐛 Debugging

### Enable Debug Logging

```bash
# Temporarily enable debug logging
nano config/config.yaml

# Change:
logging:
  level: DEBUG  # Instead of INFO

# Run script manually
source venv/bin/activate
python scripts/a1_to_a2_download.py

# Check detailed logs
tail -100 logs/a1_to_a2.log
```

### Test Individual Components

**Test DAM only:**
```bash
python -c "
import sys
sys.path.insert(0, 'scripts')
from shared.config_loader import load_config
from shared.dam_client import DAMClient

config = load_config('config/config.yaml')
dam = DAMClient(config)
campaigns = dam.search_campaigns(status='A1')
print('Found {} A1 campaigns'.format(len(campaigns)))
for c in campaigns:
    print('  -', c.get('campaign_name'), c.get('campaign_id'))
"
```

**Test Box only:**
```bash
python -c "
import sys
sys.path.insert(0, 'scripts')
from shared.config_loader import load_config
from shared.box_client import BoxClient

config = load_config('config/config.yaml')
box = BoxClient(config)
print('Box connection:', box.test_connection())
"
```

**Test Database only:**
```bash
python -c "
import sys
sys.path.insert(0, 'scripts')
from shared.config_loader import load_config
from shared.database import Database

config = load_config('config/config.yaml')
db = Database(config)
print('Database connection:', db.test_connection())
db.close()
"
```

---

## 📞 Support & Escalation

### Check Script Health

**Quick health check script:**
```bash
#!/bin/bash
# health_check.sh

echo "=== Ferrero Automation Health Check ==="
echo ""

# Check cron is scheduled
echo "Cron Jobs:"
crontab -l | grep ferrero

echo ""
echo "Recent Executions:"
tail -5 logs/cron.log

echo ""
echo "Recent Successes:"
grep "Campaign completed successfully" logs/a1_to_a2.log | tail -3

echo ""
echo "Recent Errors:"
grep -i error logs/a1_to_a2.log | tail -5

echo ""
echo "Database Records (last 24h):"
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5433 -U ferrero_user -d ferrero_tracking -c "
SELECT COUNT(*) as count, DATE(created_at) as date
FROM master_assets
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY DATE(created_at);
"
```

**Make executable and run:**
```bash
chmod +x health_check.sh
./health_check.sh
```

### If Script Stops Working

1. **Check cron is running:**
   ```bash
   service cron status  # or: systemctl status cron
   ```

2. **Check disk space:**
   ```bash
   df -h
   ```

3. **Check logs for errors:**
   ```bash
   tail -100 logs/a1_to_a2.log | grep -i error
   ```

4. **Test manually:**
   ```bash
   source venv/bin/activate
   python scripts/a1_to_a2_download.py
   ```

5. **Check database connectivity:**
   ```bash
   python scripts/test_connection.py
   ```

### Contact Information

**For script issues:**
- Check logs first: `logs/a1_to_a2.log`
- Check cron log: `logs/cron.log`
- Email: daveporter@oliver.agency

**For server issues:**
- Contact hosting provider
- Check Python 3.6 is still available
- Check cron service is running

---

## 🎯 Production Deployment Checklist

### Pre-Deployment
- [ ] Uploaded all files to server
- [ ] Created virtual environment
- [ ] Installed dependencies (requirements.txt)
- [ ] Configured .env with production credentials
- [ ] Copied Box-config.json to correct location
- [ ] Verified all paths in config.yaml

### Testing
- [ ] Ran test_connection.py - all green
- [ ] Ran a1_to_a2_download.py manually - worked
- [ ] Verified files uploaded to Box (correct folder)
- [ ] Verified database records created
- [ ] Received email notification
- [ ] Webhook sent to Make.com

### Deployment
- [ ] Setup cron job for A1→A2
- [ ] Verified cron is scheduled (crontab -l)
- [ ] Waited 5 minutes, checked cron.log
- [ ] Verified script executed automatically
- [ ] Setup A2→A3 webhook server (if ready)
- [ ] Setup webhook monitor cron

### Post-Deployment
- [ ] Monitor logs for 24 hours
- [ ] Check emails are being received
- [ ] Verify campaigns progressing through workflow
- [ ] Check database growth
- [ ] Verify Box folders being created correctly

---

## 📖 Quick Reference Commands

### Daily Operations
```bash
# Check if script is working
tail -20 logs/a1_to_a2.log

# Count processed assets today
grep "$(date +%Y-%m-%d)" logs/a1_to_a2.log | grep "Campaign completed" | wc -l

# View latest email sent
grep "Email sent" logs/a1_to_a2.log | tail -1

# Check webhook status
grep "Webhook sent" logs/a1_to_a2.log | tail -5
```

### Emergency Procedures

**Stop A1→A2 automation:**
```bash
# Remove from crontab
crontab -e
# Comment out or delete the line with a1_to_a2_download.py
```

**Stop A2→A3 webhook:**
```bash
# If running
kill $(cat webhook.pid)
rm webhook.pid
```

**Restart everything:**
```bash
# Re-add to crontab
crontab -e

# Restart webhook
cd ~/ferrero-automation/Python-Version
source venv/bin/activate
nohup python scripts/a2_to_a3_upload.py > logs/webhook.log 2>&1 &
echo $! > webhook.pid
```

---

## ✅ Success Criteria

**A1→A2 automation is working if:**
- ✅ Cron runs every 5 minutes (check cron.log)
- ✅ A1 campaigns are processed (check a1_to_a2.log)
- ✅ Files appear in Box folder 348304357505
- ✅ Database records increase
- ✅ Campaign status changes from A1 to A2
- ✅ Emails arrive at daveporter@oliver.agency
- ✅ Webhooks arrive at Make.com

---

## 📞 Support

**Documentation:**
- README.md - Quick start guide
- DEPLOYMENT.md - This file
- PYTHON_AUTOMATION_PLAN.md - Complete architecture
- PROJECT_STATUS_2025-10-30.md - Full session summary

**Logs Location:**
- Application logs: `logs/a1_to_a2.log`
- Cron logs: `logs/cron.log`
- Webhook logs: `logs/webhook.log`
- Error logs: `logs/errors.log`

**Configuration Files:**
- Main config: `config/config.yaml`
- Field mappings: `config/field_mappings.yaml`
- Environment: `.env`
- Production overrides: `config/environments/production.yaml`

---

**Deployment Version:** 1.0.0
**Python Version Required:** 3.6+
**Tested With:** Python 3.6 (server), Python 3.10 (local)
**Status:** Production Ready

---

**For additional support, refer to PROJECT_STATUS_2025-10-30.md**