e1f15ea632
Feature 1: A1→A2 Empty Folder Retry Logic - Track retry attempts (max 3) for campaigns with no master assets - Mark campaigns as permanently failed after 3 attempts - Stop processing and sending emails for permanently failed campaigns - Two new email templates: retry notification and permanent failure - Database migration adds 4 new columns to campaign_status table - Comprehensive documentation in A1_RETRY_LOGIC.md Feature 2: Orchestrator Off-Hours Cadence - Add 30 minutes to all task intervals during off-hours - Off-hours: 10 PM - 5 AM weekdays + all day Saturday/Sunday - Tasks only run at minutes 0 and 30 during off-hours - Configurable and easy to enable/disable - Daily Report (7 PM) remains unchanged Files changed: - NEW: database/migrations/003_add_a1_retry_tracking.sql - NEW: MARKDOWN_DOCS/A1_RETRY_LOGIC.md - MODIFIED: scripts/shared/database.py (added 3 methods) - MODIFIED: scripts/a1_to_a2_box_uploader.py (added retry logic) - MODIFIED: scripts/shared/notifier.py (added 2 templates) - MODIFIED: scripts/orchestrator-prod.py (added off-hours config) - MODIFIED: RUN_ORCHESTRATOR.md (added off-hours docs) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
178 lines
4 KiB
Markdown
178 lines
4 KiB
Markdown
# Running the Orchestrator
|
|
|
|
## Quick Start - Watch in Real Time
|
|
|
|
To run the orchestrator and watch it in real-time (daemon mode):
|
|
|
|
```bash
|
|
cd /Users/daveporter/Desktop/CODING-2024/Ferrero-Opentext/Python-Version
|
|
python3 scripts/orchestrator.py --daemon
|
|
```
|
|
|
|
This will:
|
|
- Run continuously in the foreground
|
|
- Check every minute for tasks that need to run
|
|
- Show all output in your terminal
|
|
- Press `Ctrl+C` to stop
|
|
|
|
---
|
|
|
|
## Other Run Modes
|
|
|
|
### Force Run All Tasks Immediately
|
|
```bash
|
|
python3 scripts/orchestrator.py --force
|
|
```
|
|
Runs all tasks right now, ignoring their schedules.
|
|
|
|
### Single Check (Cron Mode)
|
|
```bash
|
|
python3 scripts/orchestrator.py
|
|
```
|
|
Checks once, runs any due tasks, then exits. This is what cron would call.
|
|
|
|
---
|
|
|
|
## Current Task Schedule
|
|
|
|
- **A1→A2 Box Uploader**: Every 5 minutes (processes 2 campaigns)
|
|
- **A2→A3 Upload Polling**: Every 5 minutes (processes ALL files)
|
|
- **A4 Box Uploader**: Every 10 minutes
|
|
- **A4 Webhook Monitor**: Every 5 minutes
|
|
- **A5→A6 Download**: Every 5 minutes
|
|
- **B1→B2 Download**: Every 10 minutes
|
|
- **Daily Report**: Once per day at 7 PM
|
|
|
|
---
|
|
|
|
## Off-Hours Configuration
|
|
|
|
### Overview
|
|
|
|
The orchestrator automatically reduces task frequency during off-hours to minimize system load during low-activity periods.
|
|
|
|
**What changes during off-hours:**
|
|
- All tasks run less frequently (only at 0 and 30 minute marks)
|
|
- Example: A 3-minute task normally runs at minutes 0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, etc.
|
|
- During off-hours: Runs only at minutes 0 and 30 (every 30 minutes)
|
|
- Daily Report (7 PM) remains unchanged
|
|
|
|
**Off-hours definition:**
|
|
- Late night: 10 PM (22:00) to 5 AM (05:00) every day
|
|
- All day Saturday (00:00-23:59)
|
|
- All day Sunday (00:00-23:59)
|
|
|
|
### Configuration
|
|
|
|
**Location:** `scripts/orchestrator-prod.py` lines ~88-107
|
|
|
|
```python
|
|
OFF_HOURS_CONFIG = {
|
|
'enabled': True, # Set to False to disable
|
|
'extra_minutes': 30, # Minutes to add during off-hours
|
|
|
|
'late_night_start': 22, # Start hour (22 = 10 PM)
|
|
'late_night_end': 5, # End hour (5 = 5 AM)
|
|
|
|
'weekend_days': [5, 6], # Saturday=5, Sunday=6
|
|
|
|
'exempt_tasks': [
|
|
'Daily Report' # Tasks that ignore off-hours
|
|
]
|
|
}
|
|
```
|
|
|
|
### Examples
|
|
|
|
**Business Hours (Monday 2 PM):**
|
|
```
|
|
A1→A2: Runs every 3 minutes (0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, ...)
|
|
A4 Box: Runs every 10 minutes (0, 10, 20, 30, 40, 50)
|
|
```
|
|
|
|
**Off-Hours (Monday 11 PM or Saturday):**
|
|
```
|
|
A1→A2: Runs every 30 minutes (0, 30)
|
|
A4 Box: Runs every 30 minutes (0, 30)
|
|
All tasks: Only run at minutes 0 and 30
|
|
```
|
|
|
|
### Customization
|
|
|
|
#### Change off-hours timing
|
|
|
|
Edit `orchestrator-prod.py`:
|
|
|
|
```python
|
|
# Late night only from midnight to 6 AM
|
|
'late_night_start': 0,
|
|
'late_night_end': 6,
|
|
|
|
# Include only Sunday as weekend
|
|
'weekend_days': [6], # 6 = Sunday
|
|
```
|
|
|
|
#### Disable off-hours completely
|
|
|
|
```python
|
|
OFF_HOURS_CONFIG = {
|
|
'enabled': False, # Turns off all off-hours logic
|
|
# ... rest unchanged
|
|
}
|
|
```
|
|
|
|
#### Exempt specific tasks
|
|
|
|
```python
|
|
'exempt_tasks': [
|
|
'Daily Report',
|
|
'A4 Webhook Monitor' # This task will run at normal cadence even in off-hours
|
|
]
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
Check orchestrator logs to see current mode:
|
|
|
|
```bash
|
|
# Watch for mode changes
|
|
tail -f logs/orchestrator.log | grep "MODE"
|
|
|
|
# Output examples:
|
|
# Orchestrator tick: 2026-01-31 14:00:00 [NORMAL MODE]
|
|
# Orchestrator tick: 2026-01-31 22:00:00 [OFF-HOURS MODE]
|
|
# Adding 30 minutes to all task intervals
|
|
```
|
|
|
|
### Testing
|
|
|
|
```bash
|
|
# Test without affecting production
|
|
python scripts/orchestrator-prod.py --force
|
|
|
|
# Look for these log messages:
|
|
# [OFF-HOURS MODE] or [NORMAL MODE]
|
|
# "Adding 30 minutes to all task intervals"
|
|
# "Task 'A1->A2' due (off-hours: 3min + 30min cadence)"
|
|
```
|
|
|
|
---
|
|
|
|
## Logs
|
|
|
|
- **Orchestrator logs**: `logs/orchestrator.log`
|
|
- **Individual script logs**: `logs/a1_to_a2_box.log`, `logs/a2_to_a3.log`, etc.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
If you see git output instead of orchestrator output, make sure you're running:
|
|
```bash
|
|
python3 scripts/orchestrator.py --daemon
|
|
```
|
|
|
|
NOT:
|
|
```bash
|
|
git pull
|
|
```
|