ferrero-opentext/Python-Version/MARKDOWN_DOCS/A1_RETRY_LOGIC.md

A1→A2 Empty Folder Retry Logic

Preventing Infinite Error Emails for Empty Campaigns

Purpose: Avoid sending "no assets found" error emails every 3 minutes indefinitely when a campaign is set to A1 but has no master assets.

Author: Claude Code Date: January 31, 2026 Related Files:

• scripts/a1_to_a2_box_uploader.py (main script)
• scripts/shared/database.py (retry tracking methods)
• database/migrations/003_add_a1_retry_tracking.sql (schema)

---

How It Works

The Problem

Previously, when a campaign had status A1 but the Master Assets folder was empty:

• System sent error email every 3 minutes
• Campaign remained in A1 status forever
• No distinction between temporary and permanent failures
• Notification fatigue for support team

The Solution

Three-attempt retry mechanism with permanent failure tracking:

Flow:

1. Campaign in A1 status with no assets → Attempt 1 (email sent, retry_count=1)
2. Still no assets after 3 minutes → Attempt 2 (email sent, retry_count=2)
3. Still no assets after 6 minutes → Attempt 3 (email sent, retry_count=3, permanently_failed=TRUE)
4. Campaign now skipped on all future runs → Manual reset required

Database Tracking

Four new fields in campaign_status table:

• a1_retry_count (INTEGER): Number of failed attempts (0-3)
• a1_last_retry_at (TIMESTAMP): When last attempt occurred
• a1_permanently_failed (BOOLEAN): TRUE after 3 failures
• a1_failure_reason (TEXT): Why it failed (e.g., "No master assets found")

---

Configuration

Maximum Retry Attempts

Current Setting: 3 attempts

To Change: Edit /Users/nickviljoen/Desktop/Ferrero/ferrero-opentext/Python-Version/scripts/shared/database.py

def increment_a1_retry(self, campaign_id, campaign_number, campaign_name, reason):
    """..."""
    # Maximum retry attempts before marking as permanently failed
    MAX_RETRIES = 3  # CHANGE THIS NUMBER

Recommendation: Keep at 3. This allows:

• Immediate notification (attempt 1)
• Short-term retry (attempt 2 after 3 min)
• Medium-term retry (attempt 3 after 6 min)
• Permanent failure (after 9 minutes total)

---

Email Notifications

Retry Email (Attempts 1-2)

Subject: ⚠️ No Assets Found (Attempt X/3) - Campaign {name} Recipients: Error notification list Content:

• Current retry count
• Remaining attempts
• What happens next

Final Failure Email (Attempt 3)

Subject: ❌ PERMANENTLY FAILED - Campaign {name} (No Assets After 3 Attempts) Recipients: Error notification list Content:

• Campaign marked as permanently failed
• Required actions to fix
• SQL command to manually reset

---

Manual Operations

Check Campaign Retry Status

SELECT campaign_number, campaign_name, status,
       a1_retry_count, a1_last_retry_at,
       a1_permanently_failed, a1_failure_reason
FROM campaign_status
WHERE campaign_id = 'YOUR_CAMPAIGN_ID';

Reset Single Campaign

UPDATE campaign_status
SET a1_retry_count = 0,
    a1_last_retry_at = NULL,
    a1_permanently_failed = FALSE,
    a1_failure_reason = NULL
WHERE campaign_id = 'YOUR_CAMPAIGN_ID';

Or using psql command:

PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking <<EOF
UPDATE campaign_status
SET a1_retry_count = 0,
    a1_last_retry_at = NULL,
    a1_permanently_failed = FALSE,
    a1_failure_reason = NULL
WHERE campaign_id = 'YOUR_CAMPAIGN_ID';
EOF

Reset All Failed Campaigns

UPDATE campaign_status
SET a1_retry_count = 0,
    a1_last_retry_at = NULL,
    a1_permanently_failed = FALSE,
    a1_failure_reason = NULL
WHERE a1_permanently_failed = TRUE;

View All Failed Campaigns

SELECT campaign_number, campaign_name,
       a1_retry_count, a1_last_retry_at, a1_failure_reason
FROM campaign_status
WHERE a1_permanently_failed = TRUE
ORDER BY a1_last_retry_at DESC;

---

Failure Scenarios

Scenario 1: Temporary Empty Folder

What Happens:

• Attempt 1: Email sent, retry counter = 1
• Assets added to folder before attempt 2
• Next run finds assets, processes successfully
• Retry counter automatically reset to 0

Result: Problem self-resolves, minimal notifications

Scenario 2: Persistent Empty Folder

What Happens:

• Attempt 1 (0 min): Email sent, retry counter = 1
• Attempt 2 (3 min): Email sent, retry counter = 2
• Attempt 3 (6 min): Email sent, retry counter = 3
• Campaign marked permanently failed
• Processing stops, no more emails

Result: Support team alerted, infinite emails prevented

Scenario 3: Wrong Status Assignment

What Happens:

• Campaign set to A1 by mistake (no assets intended)
• Fails 3 times, marked permanently failed
• Admin realizes mistake, changes status to different value
• Campaign no longer appears in A1 search results

Result: No reset needed, campaign excluded from processing

---

Testing

Test Retry Logic

1. Create test campaign in DAM with A1 status
2. Ensure Master Assets folder is empty
3. Run A1→A2 script manually 3 times
4. Verify emails received and database state

# Run 1
python scripts/a1_to_a2_box_uploader.py --auth-pfx-v2

# Check database
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT campaign_number, a1_retry_count, a1_permanently_failed FROM campaign_status WHERE status = 'A1';"

# Run 2 (wait 3 minutes or run immediately for testing)
python scripts/a1_to_a2_box_uploader.py --auth-pfx-v2

# Check again
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT campaign_number, a1_retry_count, a1_permanently_failed FROM campaign_status WHERE status = 'A1';"

# Run 3
python scripts/a1_to_a2_box_uploader.py --auth-pfx-v2

# Verify permanently failed
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "SELECT campaign_number, a1_retry_count, a1_permanently_failed, a1_failure_reason FROM campaign_status WHERE a1_permanently_failed = TRUE;"

Test Reset Logic

# Reset the test campaign
PGPASSWORD=ferrero_pass_2025 psql -h localhost -p 5437 -U ferrero_user -d ferrero_tracking -c "UPDATE campaign_status SET a1_retry_count = 0, a1_permanently_failed = FALSE WHERE campaign_number = 'TEST_CAMPAIGN';"

# Run again
python scripts/a1_to_a2_box_uploader.py --auth-pfx-v2

# Verify it retries

---

Monitoring

Dashboard Query: Current Retry Status

SELECT
    COUNT(*) FILTER (WHERE a1_retry_count = 0) as "No Issues",
    COUNT(*) FILTER (WHERE a1_retry_count = 1) as "Attempt 1",
    COUNT(*) FILTER (WHERE a1_retry_count = 2) as "Attempt 2",
    COUNT(*) FILTER (WHERE a1_retry_count >= 3) as "Permanently Failed"
FROM campaign_status
WHERE status = 'A1';

Alert Query: Campaigns Near Failure

SELECT campaign_number, campaign_name, a1_retry_count, a1_last_retry_at
FROM campaign_status
WHERE status = 'A1'
  AND a1_retry_count >= 2
  AND a1_permanently_failed = FALSE
ORDER BY a1_retry_count DESC, a1_last_retry_at DESC;

---

Troubleshooting

Q: Campaign keeps failing even after adding assets

A: Check if campaign was marked permanently failed. Reset using SQL command above.

Q: Want to change from 3 to 5 retry attempts

A: Edit MAX_RETRIES = 3 in database.py line ~567. Also update email templates to reflect new maximum.

Q: How to disable retry logic completely?

A: Not recommended, but you can:

1. Set MAX_RETRIES = 999 (effectively infinite)
2. Or revert to old a1_to_a2_no_assets template without retry tracking

Q: Can I set different retry counts for different campaigns?

A: No, it's a global setting. All campaigns use same MAX_RETRIES value.

Q: What if I want to delete permanently failed campaigns from database?

A: Don't delete. Instead, change their status to something other than A1. They'll be excluded from processing automatically.

---

Future Enhancements

Potential improvements for future versions:

1. Configurable retry timing:

   • Instead of relying on cron frequency (3 min)
   • Check a1_last_retry_at and skip if too recent
   • Allow exponential backoff (3 min, 10 min, 30 min)

2. Campaign-specific retry limits:

   • Add optional a1_max_retries column
   • Allow different campaigns to have different thresholds
   • Default to global MAX_RETRIES if not set

3. Automatic cleanup:

   • After 30 days, auto-reset permanently failed campaigns
   • Or send weekly digest of stuck campaigns

4. Webhook notifications:

   • Send to external system when campaign permanently fails
   • Integrate with ticketing system

5. Admin UI:

   • Web interface to view/reset retry status
   • Bulk reset operations

---

Code Locations

Quick reference for developers:

Component	File	Line Range
Retry check logic	a1_to_a2_box_uploader.py	~176-186
Empty folder detection	a1_to_a2_box_uploader.py	~193-231
Success reset	a1_to_a2_box_uploader.py	~354-356
get_a1_retry_status()	database.py	~522-558
increment_a1_retry()	database.py	~560-620
reset_a1_retry()	database.py	~622-655
Email templates	notifier.py	~593-687
Database migration	migrations/003_add_a1_retry_tracking.sql	All

---

Change Log

January 31, 2026:

• Initial implementation
• 3-attempt retry mechanism
• Permanent failure tracking
• Two new email templates
• This documentation created

Future updates will be logged here.