creative-x-ferrero/creativex-automation

CreativeX Automation Service

Automated Box.com folder monitoring service that:

• Monitors Box "Ferrero-In" folder for new creative files
• Automatically uploads to CreativeX for quality analysis
• Polls for completion (24+ hours)
• Extracts scores and stores in PostgreSQL database
• Sends formatted email results to file uploader

Production Deployment Summary

✅ Service Installed and Running

Server: /opt/ferrero-creativex/creative-x-ferrero/creativex-automation Systemd Service: creativex-automation.service Status: sudo systemctl status creativex-automation

Box Folders:

• Ferrero-In: 363284027140 (monitored for new files)
• Processing: 363306582612 (files being analyzed)
• Processed: 363307501826 (completed files)

Database: PostgreSQL localhost:5437/ferrero_tracking Table: creativex_scores (existing)

Email: SMTP via Mailgun Sender: TWIST-UK-SERVER@oliver.agency Recipients: File uploader + daveporter@oliver.agency

API: Production CreativeX (https://api.creativex.com/api/v3) Supported: 93 brands, 44+ channels

Timing:

• Box scan: Every 5 minutes
• Status check: Every 30 minutes
• Queue cleanup: Every 48 hours

Management:

sudo systemctl start|stop|restart|status creativex-automation
sudo journalctl -u creativex-automation -f
tail -f data/logs/creativex_automation.log

---

Quick Start

1. Setup

cd /Users/daveporter/Desktop/CODING-2024/CREATIVE-X/creativex-automation

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
# Update .env with your credentials
nano .env

2. Configuration

Required in .env:

• CreativeX API credentials (from CREATIVE-X/.env)
• Database credentials (from Ferrero-Opentext/.env)
• Mailgun SMTP credentials (from Ferrero-Opentext/.env)
• Box folder IDs

Required file:

• config/43984435_llhzqo5z_config.json - Box JWT config (copy from Ferrero-Opentext)

3. Test Configuration

# Verify configuration
python service.py --dry-run

# Expected output:
# DRY RUN MODE - Configuration Valid ✓

4. Test Box Folder Scan

# Scan Box folder once
python service.py --scan-once

# Expected output:
# Found N new file(s)
#   - filename1.mp4 (uploader: user@example.com)

5. Run Service

# Run manually for testing
python service.py

# Watch logs in another terminal
tail -f data/logs/creativex_automation.log

# Stop with Ctrl+C

---

How It Works

Workflow

Box Ferrero-In (363284027140)
  ↓ User drops file
Box Monitor (scans every 5 min)
  ↓ Detects new file, gets uploader email
Upload Processor (immediate)
  ↓ Parse filename, validate, upload to CreativeX
  ↓ Move file to Ferrero-In/Processing
  ↓ Send "Upload Started" email
Status Poller (checks every 30 min)
  ↓ Polls until completed (24+ hours)
Result Handler (immediate when complete)
  ↓ Extract scores, store in DB
  ↓ Send "Analysis Complete" email
  ↓ Move file from Processing to Processed
✅ Complete!

Box Folder Structure

Ferrero-In (363284027140)

• Main folder where users drop files
• Service monitors this folder every 5 minutes
• New files detected here

Ferrero-In/Processing (auto-created)

• Files currently being analyzed by CreativeX
• Moved here immediately after upload
• Stays here during 1-24 hour processing

Ferrero-In/Processed (auto-created)

• Completed files
• Moved here after analysis complete
• Includes files with results in database

Service Timing

Component	Interval	Purpose
Box Monitor	5 minutes	Detect new files
Upload Processor	Immediate	Upload to CreativeX
Status Poller	30 minutes	Check for completion
Result Handler	Immediate	Email results

File Lifecycle

1. User drops file in Box Ferrero-In folder
2. Detected within 5 minutes by Box monitor
3. Uploaded to CreativeX within 1 minute
4. 📧 "Upload Started" email sent to uploader + daveporter@oliver.agency
5. Processed by CreativeX (1-48 hours, typically 24 hours)
   • File stays in Ferrero-In during processing
   • Service polls every 30 minutes for completion
6. 📧 "Analysis Complete" email sent with scores when ready
7. File moved to Ferrero-In/Processed subfolder
8. Results stored in creativex_scores database table

Email Notifications

3 email types sent:

1. 🚀 Upload Started (immediate)

   • Confirms file received and uploaded to CreativeX
   • Includes: Brand, Market, Channel, Request ID
   • Sent when: Upload succeeds

2. ✅ Analysis Complete (1-24 hours later)

   • Full score breakdown with guidelines
   • Scorecard URL for PDF report
   • Sent when: CreativeX processing completes

3. ❌ Upload Failed (immediate)

   • Error message and details
   • File stays in Box for manual review
   • Sent when: Validation or upload fails

Multiple Files

You can drop multiple files simultaneously:

• Service processes ALL detected files immediately
• Each file gets its own request_id
• All process in parallel in CreativeX
• Each gets separate "started" and "complete" emails
• Files move to Processed individually when complete

---

Directory Structure

creativex-automation/
├── service.py                    # Main entry point
├── config.py                     # Configuration loader
├── requirements.txt              # Python dependencies
├── .env                         # Environment variables
├── data.json                    # Ferrero naming convention
├── mappings.json                # Ferrero → CreativeX mappings
│
├── automation/                  # Automation components
│   ├── box_monitor.py          # Monitor Box folder
│   ├── upload_processor.py     # Upload to CreativeX
│   ├── status_poller.py        # Poll CreativeX status
│   ├── result_handler.py       # Extract scores, store, email
│   ├── orchestrator.py         # Service coordination
│   └── processing_queue.py     # Queue management
│
├── core/                        # CreativeX integration (from CREATIVE-X)
│   ├── api_client.py           # CreativeX API client
│   ├── filename_parser.py      # Parse Ferrero filenames
│   ├── mapping_resolver.py     # Map Ferrero → CreativeX
│   ├── validators.py           # File validation
│   └── data_loader.py          # Load data.json
│
├── shared/                      # Shared components (from Ferrero-Opentext)
│   ├── box_client.py           # Box SDK JWT client
│   ├── database.py             # PostgreSQL client
│   └── notifier.py             # Email sender (with new templates)
│
├── utils/                       # Utilities
│   ├── logger.py               # Logging setup
│   └── file_handler.py         # File operations
│
├── data/                        # Runtime data
│   ├── processing_queue.json   # Processing queue state
│   └── logs/                   # Service logs
│
├── config/                      # Configuration files
│   └── 43984435_llhzqo5z_config.json  # Box JWT config
│
└── deployment/                  # Deployment configs
    └── systemd/                 # Systemd service unit
        └── creativex-automation.service

---

Database Schema

Existing Table: creativex_scores

Service uses existing table from Ferrero-Opentext project:

CREATE TABLE creativex_scores (
    id SERIAL PRIMARY KEY,
    filename VARCHAR(500) NOT NULL,
    box_file_id VARCHAR(255),
    creativex_id VARCHAR(255),           -- CreativeX request ID
    creativex_url TEXT,                  -- Scorecard URL
    quality_score VARCHAR(50),           -- Percentage e.g., "67"
    full_extraction_data JSONB,          -- Complete CreativeX API response
    tracking_id VARCHAR(6),              -- Optional (not used by automation)
    extracted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    status VARCHAR(50) DEFAULT 'active',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Data Storage:

• Uses existing database.store_creativex_score() function
• Maintains compatibility with other tools
• Full CreativeX API response stored in JSONB column

---

Email Templates

Success Email: creativex_upload_complete

Styled with purple CreativeX branding (#9c27b0), includes:

• Filename, brand, market, channel, placement
• Quality score with large display (e.g., "67%")
• Quality tier with color coding
• Guidelines breakdown (✅/❌ with weights)
• Scorecard URL for full PDF report
• Processing time
• Request ID

Recipients: Box uploader + daveporter@oliver.agency

Error Email: creativex_upload_failed

Red error styling, includes:

• Filename
• Error message
• Action required
• Contact information

Recipients: Box uploader + daveporter@oliver.agency

---

Testing

Test Configuration

# 1. Test config loads correctly
python service.py --dry-run

# Expected: "Configuration Valid ✓"

Test Box Connection

# 2. Test Box folder access
python service.py --scan-once

# Expected: Lists files in Ferrero-In folder

Test Upload (Manual)

# 3. Upload a test file manually to verify components
cd ..
source venv/bin/activate

# Use existing upload script
python scripts/upload.py /path/to/test_file.mp4

# Check status
python scripts/check_status.py --detailed

Test Full Service

# 4. Run service and drop test file in Box
python service.py

# In another terminal, watch logs
tail -f data/logs/creativex_automation.log

# Drop test file in Box Ferrero-In folder
# Watch logs for:
# - File detection
# - Upload to CreativeX
# - Status checks (every 30 min)
# - Completion
# - Email sent
# - File moved to Processed

Verify Database

# Check database for results
psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking

SELECT filename, creativex_id, quality_score, creativex_url
FROM creativex_scores
ORDER BY created_at DESC
LIMIT 5;

---

Deployment (Production)

Setup on Server (Step-by-Step)

Step 1: SSH to Server

ssh user@your-server

Step 2: Navigate to Project Directory

cd /opt/ferrero-creativex

Step 3: Clone Repository

git clone git@bitbucket.org:zlalani/creative-x-ferrero.git
cd creative-x-ferrero/creativex-automation

Step 4: Create Virtual Environment

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Step 5: Copy Database & SMTP Credentials

# Get values from Ferrero-Opentext project
cat /opt/ferrero-automation/Python-Version/.env | grep -E "DB_|SMTP_|SENDER_EMAIL"

# Copy those values plus add CreativeX credentials
cp .env.template .env
nano .env

Update .env with:

• Database credentials (from Ferrero-Opentext)
• SMTP credentials (from Ferrero-Opentext)
• CreativeX production API credentials
• Box folder IDs (hardcoded):

  BOX_FERRERO_IN_FOLDER_ID=363284027140
  BOX_PROCESSING_FOLDER_ID=363306582612
  BOX_PROCESSED_FOLDER_ID=363307501826
  

Step 6: Copy Box JWT Config

mkdir -p config
# Copy JWT config to config/43984435_llhzqo5z_config.json

Step 7: Test Configuration

python service.py --dry-run

Expected: DRY RUN MODE - Configuration Valid ✓

Step 8: Test Box Connection

python service.py --scan-once

Expected: Found N new file(s) or Found 0 new file(s)

Install as Systemd Service

Step 9: Copy and Configure Service Unit

sudo cp deployment/systemd/creativex-automation.service /etc/systemd/system/
sudo nano /etc/systemd/system/creativex-automation.service

Update these lines:

User=daveporter
Group=daveporter
WorkingDirectory=/opt/ferrero-creativex/creative-x-ferrero/creativex-automation
ExecStart=/opt/ferrero-creativex/creative-x-ferrero/creativex-automation/venv/bin/python service.py
ReadWritePaths=/opt/ferrero-creativex/creative-x-ferrero/creativex-automation/data /tmp/creativex-automation

Step 10: Enable and Start Service

# Reload systemd
sudo systemctl daemon-reload

# Enable auto-start on boot
sudo systemctl enable creativex-automation

# Start service
sudo systemctl start creativex-automation

# Check status
sudo systemctl status creativex-automation

Expected: Active: active (running)

Step 11: Verify Service is Working

# View live logs
sudo journalctl -u creativex-automation -f

# Or view service log file
tail -f /opt/ferrero-creativex/creative-x-ferrero/creativex-automation/data/logs/creativex_automation.log

Service Management

Start/Stop/Restart:

# Start service
sudo systemctl start creativex-automation

# Stop service
sudo systemctl stop creativex-automation

# Restart service
sudo systemctl restart creativex-automation

# Check status
sudo systemctl status creativex-automation

# Enable auto-start on boot
sudo systemctl enable creativex-automation

# Disable auto-start
sudo systemctl disable creativex-automation

View Logs:

# Live logs (systemd journal)
sudo journalctl -u creativex-automation -f

# Last 100 lines
sudo journalctl -u creativex-automation -n 100

# Service log file (rotating)
tail -f data/logs/creativex_automation.log

# Last 50 lines from file
tail -50 data/logs/creativex_automation.log

# Search logs for errors
grep ERROR data/logs/creativex_automation.log

Check Processing Queue:

# View queue
cat data/processing_queue.json | python -m json.tool

# Count files by status
cat data/processing_queue.json | python -c "
import json, sys
data = json.load(sys.stdin)
files = data.get('files', {})
print(f'Total: {len(files)}')
for status in ['pending', 'uploading', 'processing', 'completed', 'failed']:
    count = sum(1 for f in files.values() if f['status'] == status)
    print(f'{status.capitalize()}: {count}')
"

Update Service After Code Changes:

# Pull latest changes
git pull origin master

# Restart service
sudo systemctl restart creativex-automation

# Watch logs
sudo journalctl -u creativex-automation -f

---

Monitoring

Check Service Health

# Service status
sudo systemctl status creativex-automation

# Recent logs
journalctl -u creativex-automation -n 50

# Log file
tail -f data/logs/creativex_automation.log

Check Processing Queue

# View queue state
cat data/processing_queue.json | python -m json.tool

# Count by status
cat data/processing_queue.json | python -c "
import json, sys
data = json.load(sys.stdin)
files = data.get('files', {})
print(f'Total: {len(files)}')
print(f'Pending: {sum(1 for f in files.values() if f[\"status\"] == \"pending\")}')
print(f'Processing: {sum(1 for f in files.values() if f[\"status\"] == \"processing\")}')
print(f'Completed: {sum(1 for f in files.values() if f[\"status\"] == \"completed\")}')
print(f'Failed: {sum(1 for f in files.values() if f[\"status\"] == \"failed\")}')
"

Query Database

# Recent uploads
psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT filename, creativex_id, quality_score, status, extracted_at
FROM creativex_scores
ORDER BY extracted_at DESC
LIMIT 10;
"

# Check for stuck processing
psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "
SELECT COUNT(*) as stuck_count
FROM creativex_scores
WHERE status = 'active'
  AND extracted_at < NOW() - INTERVAL '48 hours';
"

---

Troubleshooting

Service Won't Start

Check logs:

journalctl -u creativex-automation -n 100
tail -f data/logs/creativex_automation.log

Common issues:

• Missing .env file or credentials
• Box JWT config file not found
• Database connection failed
• Python dependencies not installed

Files Not Being Detected

Check:

# Test Box connection
python service.py --scan-once

# Verify folder ID
echo $BOX_FERRERO_IN_FOLDER_ID

# Check Box permissions
# Ensure service account has access to folder

Uploads Failing

Check logs for:

• Filename parsing errors (invalid format)
• Unsupported brand/channel (not in mappings.json)
• File too large (>500MB)
• CreativeX API errors

Verify:

# Test upload manually
cd ..
python scripts/upload.py --dry-run test_file.mp4

Emails Not Sending

Check:

• SMTP credentials in .env
• Mailgun account status
• Recipient email addresses valid
• Check logs for email errors

Test:

# Test email sending
python -c "
from shared.notifier import Notifier
config = {'notifications': {'enabled': True, 'smtp': {...}}}
notifier = Notifier(config)
notifier.send_email('test', ['daveporter@oliver.agency'], {'test': 'data'})
"

Database Issues

Check connection:

psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\conninfo"

Verify table exists:

psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "\d creativex_scores"

---

Configuration Reference

Environment Variables

Service:

• SERVICE_NAME - Service identifier
• SERVICE_CHECK_INTERVAL - Box scan interval (seconds, default: 300)
• STATUS_CHECK_INTERVAL - CreativeX status check interval (seconds, default: 1800)

Box:

• BOX_JWT_CONFIG_PATH - Path to Box JWT config file
• BOX_FERRERO_IN_FOLDER_ID - Folder to monitor (363284027140)
• BOX_PROCESSED_FOLDER_ID - Processed files folder (auto-created if empty)

CreativeX:

• CREATIVEX_API_BASE_URL - API endpoint
• CREATIVEX_ACCESS_TOKEN - API token
• DATA_JSON_PATH - Path to data.json
• MAPPINGS_JSON_PATH - Path to mappings.json

Database:

• DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD

Email:

• SMTP_SERVER, SMTP_PORT, SMTP_USER, SMTP_PASSWORD
• SENDER_EMAIL, CC_EMAIL, ERROR_EMAIL

Processing:

• MAX_PROCESSING_HOURS - Timeout for CreativeX processing (default: 48)
• MAX_RETRIES - Retry count for failed operations (default: 3)
• TEMP_DIR - Temporary file storage

Logging:

• LOG_LEVEL - INFO, DEBUG, WARNING, ERROR
• LOG_MAX_BYTES - Max log file size (default: 10MB)
• LOG_BACKUP_COUNT - Number of backup logs (default: 28)

---

Supported Brands

See mappings.json for complete list. Currently supports 93 Ferrero brands including:

• Nutella family (11 variants)
• Kinder family (60+ variants)
• Ferrero Rocher family (5 variants)
• Raffaello family (3 variants)
• Tic Tac family (4 variants)
• Plus: Mon Chéri, Pocket Coffee, Estathe, and more

---

Supported Channels

44+ social media channels including:

• Facebook (15 placements)
• Instagram (11 placements)
• YouTube (6 formats)
• TikTok, Snapchat, Pinterest, Twitter, Amazon, DV360

---

Development

Adding New Email Templates

Edit shared/notifier.py, add to templates dictionary:

'new_template_name': {
    'subject': "Subject with {variable}",
    'html': """
        <div style="font-family: Arial, sans-serif; max-width: 900px; margin: 0 auto;">
            <!-- Template HTML -->
            {{ variable }}
        </div>
    """
}

Adding New Brands

When CreativeX adds new brands:

# 1. Query dimensions
cd ..
python get_dimensions.py | grep "brands" -A 50

# 2. Update mappings.json
nano mappings.json

# 3. Test
cd creativex-automation
python service.py --dry-run

---

Support

Issues:

• Check logs: data/logs/creativex_automation.log
• Check queue: data/processing_queue.json
• Check database: Query creativex_scores table
• Contact: daveporter@oliver.agency

Documentation:

• Main CreativeX docs: ../README.md
• Mappings guide: ../MAPPINGS_GUIDE.md
• Service plan: Plan document in .claude/plans/

---

Security

Credentials:

• All credentials in .env (git-ignored)
• Box JWT private key in separate config file
• Database password not in code
• SMTP credentials secure

File Handling:

• Downloaded to temp directory
• Cleaned up after processing
• Validated before upload
• Sanitized filenames

Email:

• Validates email addresses
• Sanitizes template data
• Uses secure SMTP with TLS

---

Performance

Expected Resource Usage:

• CPU: Low (mostly I/O waiting)
• Memory: ~100-200MB
• Disk: Minimal (temp files cleaned up)
• Network: Moderate (file uploads/downloads)

Throughput:

• ~100 files/day with current timing
• Can process unlimited files concurrently (CreativeX handles queueing)

---

License

© 2026 Ferrero. All rights reserved.

---

Service Ready! 🚀

Start with: python service.py