Commit graph

22 commits

Author SHA1 Message Date
DJP
e56ae795eb Complete production deployment documentation
Added:
- DEPLOYMENT.md: Complete production deployment guide
- Updated README.md: Step-by-step server setup instructions
- Production deployment summary with all server details
- Service management commands (systemd)
- Monitoring and troubleshooting guides
- Queue cleanup automation (48 hours)

Production Configuration:
- Server: /opt/ferrero-creativex/creative-x-ferrero/creativex-automation
- Systemd service installed and running
- Box folders hardcoded: 363284027140, 363306582612, 363307501826
- Database: localhost:5437/ferrero_tracking
- Email: TWIST-UK-SERVER@oliver.agency (verified sender)
- API: Production CreativeX with 93 brands

Service Features:
- Auto-start on boot (systemd)
- Auto-restart on failure
- Rotating logs (10MB, 28 backups)
- Queue cleanup after 48 hours
- Full email notifications (upload started, complete, failed)
- 3-folder workflow (Ferrero-In → Processing → Processed)

Tested and Validated:
- Box connection working
- Database connection working
- File upload to CreativeX working
- Email delivery working
- File movement between folders working
- Queue management working

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-02-03 16:05:13 -05:00
DJP
fde6a32c44 Add automatic queue cleanup for old files
Features:
- Remove completed/failed files from queue after 48 hours
- Runs automatically every service cycle (5 min)
- Prevents queue from growing indefinitely
- Logs cleanup actions

Cleanup logic:
- Only removes files with status: completed or failed
- Checks completed_at timestamp
- Removes if older than 48 hours
- Saves queue after cleanup

Example: File completes at 10 AM Monday, removed from queue at 10 AM Wednesday

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-30 16:07:52 -05:00
DJP
b0b3ef21b1 Add missing typing import for Python compatibility
Fix: Import Dict from typing module
Issue: NameError on older Python versions
Result: Works on Python 3.8+ on production servers

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-30 14:41:45 -05:00
DJP
8c881bfd17 Update gitignore for runtime and working files
- Ignore processing_queue.json (runtime state)
- Ignore .eml files (email downloads)
- Ignore brand_mappings_*.json (working files)

These files are generated at runtime or for reference only

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-30 14:18:14 -05:00
DJP
b6f4178667 Update to use production CreativeX API
Changed from staging to production API:
- URL: https://api.creativex.com/api/v3 (was staging-api)
- Token: Production token
- Result: Full guideline data (9+ guidelines vs 2 in staging)

Production has complete Ferrero scoring with all guidelines
Staging only has simplified test data

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-30 09:12:30 -05:00
DJP
a6db820406 Show ALL scores and guidelines in completion email
Changes:
- Extract all scores from CreativeX response (not just default)
- Show each score with all its guidelines in email
- Visual distinction: Default score = purple, others = gray
- Display guideline count for each score
- Show CQS badge if applicable

Email now includes:
- All score types (Ferrero: Creative Quality Rate, Custom Score, etc.)
- All guidelines for each score (not just first 2)
- Complete breakdown matching --detailed CLI output

Example: If response has 2 scores with 9 guidelines each,
email will show both scores with all 18 guidelines total

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-30 08:18:59 -05:00
DJP
7f14bd7883 Fix uploader_email KeyError in completion handler
Bug: Status checks were failing with KeyError: 'uploader_email'
Fix: Changed to 'box_uploader_email' to match queue data structure
Impact: Completion emails and file moves will now work correctly

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 16:01:23 -05:00
DJP
352bdcdb3d Fix Box folder creation - remove inline comments from .env
Issue: Inline comments in .env were being read as folder IDs
Fix: Comment out BOX_PROCESSING_FOLDER_ID and BOX_PROCESSED_FOLDER_ID
Result: Folders will be auto-created on service start

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 12:32:41 -05:00
DJP
6a9a136b86 Fix sender email to use verified Mailgun address
- Change from CREATIVEX-AUTOMATION@oliver.agency to TWIST-UK-SERVER@oliver.agency
- TWIST-UK-SERVER is the verified sender in Mailgun
- Matches Ferrero-Opentext configuration
- Add .env.template for reference

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 12:28:49 -05:00
DJP
583711cfe3 Add upload started email and 3-folder workflow
Features:
- Send email when upload starts (immediate confirmation)
- Move files through 3 folders: Ferrero-In → Processing → Processed
- Auto-create Processing and Processed subfolders
- Keep Ferrero-In clean (only new files waiting)

Workflow:
1. User drops file in Ferrero-In
2. Service uploads to CreativeX
3. File moved to Processing folder
4. Email sent: Upload Started
5. Service polls every 30 min
6. When complete: Email sent with scores
7. File moved to Processed folder

Email templates:
- Upload Started: Blue header, shows brand/channel/request_id
- Analysis Complete: Purple header, shows scores/guidelines
- Upload Failed: Red header, shows error

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 12:12:28 -05:00
DJP
40b6f91d4b Add service status documentation
- Documented service completion status
- Box authorization issue identified and documented
- Next steps for local testing and production deployment

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 11:35:58 -05:00
DJP
4149cd1f45 Add Box SDK JWT support for Python 3.14
- Install boxsdk[jwt] with cryptography dependencies
- Service initializes successfully
- All components tested ✓

Note: Box JWT config (43984435_llhzqo5z_config.json) must be placed
in config/ directory (gitignored for security)

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 11:35:05 -05:00
DJP
9c4e8d8e23 Fix Python 3.14 compatibility - Update to psycopg3
Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 11:29:19 -05:00
DJP
b20119b383 Add complete mapping system and automated Box.com monitoring service
Major Features:
- Complete Ferrero ↔ CreativeX mapping system with 93 brands
- Automated Box.com folder monitoring service
- Email notifications with score breakdowns
- Database integration for result storage

Mapping System (v2.0.0):
- mappings.json: 93 brand mappings, 44+ channel mappings
- core/mapping_resolver.py: Translates Ferrero codes to CreativeX format
- scripts/validate_mappings.py: Validation tool for brand/channel support
- scripts/generate_brand_mappings.py: Auto-mapping tool
- scripts/download_reports.py: Scorecard PDF download tool
- Updated scripts/upload.py: Integrated mapping validation
- Updated scripts/check_status.py: Added detailed score display with guidelines

Documentation:
- Updated README.md: Complete user guide with mapping system
- Updated STATUS.md: Production-ready status with test results
- MAPPINGS_GUIDE.md: Complete mapping documentation
- MAPPING_IMPLEMENTATION.md: Implementation summary
- BRAND_MAPPINGS_REVIEW.md: Brand mapping validation guide
- PRODUCTION_BRANDS_SUMMARY.md: Production brand catalog
- PRODUCTION_MAPPING_COMPLETE.md: Mapping completion summary

Automation Service (New):
- creativex-automation/: Complete automated Box monitoring service
- Monitors Box Ferrero-In folder (363284027140) for new files
- Automatically uploads to CreativeX
- Polls for completion (30 min intervals)
- Extracts scores and stores in PostgreSQL creativex_scores table
- Sends formatted emails to file uploader + daveporter@oliver.agency
- Moves processed files to Processed subfolder

Service Components:
- automation/box_monitor.py: Box folder monitoring with uploader detection
- automation/upload_processor.py: CreativeX upload integration
- automation/status_poller.py: CreativeX status polling
- automation/result_handler.py: Score extraction and email sending
- automation/orchestrator.py: Service coordination
- automation/processing_queue.py: JSON-based processing queue
- service.py: Main service entry point
- config.py: Service configuration loader
- requirements.txt: All dependencies
- deployment/systemd/: Systemd service unit file
- Updated shared/notifier.py: Added creativex_upload_complete and creativex_upload_failed templates

Testing:
- Supports --dry-run mode for configuration testing
- Supports --scan-once mode for Box folder testing
- Manual run mode for development/testing
- Comprehensive logging with rotation (10MB, 28 backups)

Database Integration:
- Uses existing creativex_scores table (no migrations needed)
- Compatible with existing Ferrero-Opentext workflows
- Stores full CreativeX API responses in JSONB

Email Templates:
- Matches Ferrero-Opentext styling (#9c27b0 purple for CreativeX)
- Includes score, tier, guidelines breakdown, scorecard URL
- Recipients: Box uploader + CC to daveporter@oliver.agency

Deployment:
- Runs locally for dev/testing
- Systemd service for production
- Auto-restart on failure
- Complete documentation in creativex-automation/README.md

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
2026-01-29 09:51:16 -05:00
DJP
b6af9d79d6 Add comprehensive project status document and dimensions query tool
STATUS.md includes:
- Complete project summary
- What's been built and tested
- Current blocker (staging environment has test brands only)
- Exact next steps when real environment available
- Performance metrics and code quality notes
- Quick resume commands for future work

get_dimensions.py:
- Utility to query /dimensions endpoint
- Shows available brands, markets, channels
- Helps validate environment before uploads

Ready to resume when proper staging/production environment is available.
2026-01-09 14:55:08 -05:00
DJP
78e19ca82d Add error status handling and detailed error logging
Changes:
- Detect 'error' status from API
- Mark upload as failed when error status received
- Display full error response for debugging
- Log error details to help diagnose issues
2026-01-09 14:50:36 -05:00
DJP
3f8351dff6 Fix: Correct preflight payload structure
The API expects:
- name, brand_name, market_name, channel at top level
- creatives array with only source_url

Changes:
- Moved name and channel to top level
- Changed asset_url to source_url in creatives
- Updated API client validation to match

This fixes the 'channel is missing, name is missing, source_url is missing' error.
2026-01-09 14:46:41 -05:00
DJP
7b506ba449 Fix: Update preflight payload to match API requirements
The CreativeX API expects specific field names and structure:
- brand_name (not brand)
- market_name (not market)
- creatives array containing asset details

Changes:
- Updated preflight metadata structure in upload.py
- Updated create_preflight validation in api_client.py
- Updated state_manager to use correct field names
- Payload now includes creatives array with name, asset_url, channel

This fixes the 'brand_name, market_name, creatives are missing' error.
2026-01-09 14:45:26 -05:00
DJP
dc0a6b741c Fix: Handle presigned URL returned as plain text
The API returns the presigned URL as plain text, not JSON.

Changes:
- Updated _make_request to return text directly if it's a URL
- Updated get_presigned_url to handle string URLs and dict responses
- Better logging with longer URL preview (80 chars)

This fixes the 'Unexpected response format' error.
2026-01-09 14:43:24 -05:00
DJP
ccb5203948 Fix: Pass filename parameter to presigned_url endpoint
The CreativeX API requires a 'filename' parameter when requesting presigned URLs.

Changes:
- Updated get_presigned_url() to accept filename parameter
- Pass filename to API as query parameter
- Updated upload script to pass filename when requesting URL
- Updated test_connection to use test filename

This fixes the 'filename is missing' error.
2026-01-09 14:42:23 -05:00
DJP
f19f34ea9d Update requirements.txt: Remove pandas/openpyxl for Python 3.14 compatibility
- Removed pandas and openpyxl (optional export functionality)
- Kept core dependencies: requests, urllib3, python-dotenv, colorama
- Scripts are fully functional without data export features
2026-01-09 14:34:56 -05:00
DJP
72e9b54ff6 Initial commit: CreativeX API integration for Ferrero assets
- Ferrero filename parser with NEW format support
- CreativeX API client with retry logic
- State persistence with resume capability
- Upload and status checking scripts
- Comprehensive documentation
- Virtual environment support
2026-01-09 14:33:00 -05:00