json-parser-twist-metaserver/OLD

JSON Workflow Processor (Legacy Versions)

⚠️ Note: This folder contains older versions. Use the main json_workflow_processor-hybrid-protected.py for production.

Evolution History

This folder contains the development evolution of the JSON Workflow Processor:

Version History

1. Basic Version (json_workflow_processor.py) - Single-threaded processing
2. Local Version (json_workflow_processor-local.py) - Development testing with local paths
3. Batch Version (json_workflow_processor-batch.py) - Added concurrent processing
4. Reporting Version (json_workflow_processor-reporting.py) - Added daily statistics and email reports
5. Startup Scan Version (json_workflow_processor-startup-scan.py) - Added crash recovery
6. Hybrid Version (json_workflow_processor-hybrid.py) - Added periodic scanning backup
7. Protected Hybrid (in main folder) - ⭐ CURRENT PRODUCTION VERSION

Core Features (All Versions)

• Dynamic hot folder monitoring - Watches /data/PRODUCTION/JSON/[CLIENT_NAME]/ recursively
  • Handles hundreds of client folders that can be added dynamically
  • Structure: /data/PRODUCTION/JSON/CLIENTNAME/file.json
• JSON validation - Parses and validates JSON structure
• Smart routing system:
  • Special clients by ClientCode:
    • RECKITTBENCKISER → Monday RB + JSON_STORE/RECKITTBENCKISER/
    • RANK → Monday Rank + JSON_STORE/RANK/
  • Celtra projects by JobCategory:
    • Only eligible clients: CIBC, OLIVER, ADIDAS, PAYPAL, RECKITTBENCKISER, BAYER, 3M, RANK
    • If JobCategory == "Celtra" → Celtra - Create_Rename - Project_Design File + JSON_STORE/[CLIENT]/
  • All other clients → JSON_STORE/[CLIENT_NAME]/ only
• File processing - Original files deleted after successful copying
• Error notifications - Email alerts for JSON parsing failures
• Activity logging - Comprehensive audit trail

Enhanced Features (Later Versions)

• Concurrent processing - ThreadPoolExecutor for 5-10 simultaneous files
• Daily reporting - Automated email statistics at midnight
• Startup scanning - Crash recovery processes existing files
• Periodic scanning - 60s backup scans for missed files
• Failed file handling - Problematic files moved to failed folder
• Protected scanning - Prevents overlapping scans
• Client folder logging - Shows client names in log messages
• Permission handling - Automatic 777 permissions for root users

Installation

pip install -r requirements.txt

Configuration

Edit the Config class in json_workflow_processor.py to adjust:

• File paths
• Email settings
• Monitoring intervals
• Client destinations

Usage

python json_workflow_processor.py

The script will run continuously, monitoring the hot folder for new JSON files.

Routing Examples

RECKITTBENCKISER client with normal project:

• File: /data/PRODUCTION/JSON/RECKITTBENCKISER/project.json
• ClientCode: "RECKITTBENCKISER"
• Result: Copied to both Monday RB/ and JSON_STORE/RECKITTBENCKISER/

ADIDAS client with Celtra project:

• File: /data/PRODUCTION/JSON/ADIDAS/campaign.json
• JobCategory: "Celtra"
• Result: Copied to both Celtra - Create_Rename - Project_Design File/ and JSON_STORE/ADIDAS/

Regular client:

• File: /data/PRODUCTION/JSON/NIKE/design.json
• Result: Copied to JSON_STORE/NIKE/ only

JSON Structure Expected

The script expects JSON files with this structure:

{
  "JobSpecification": {
    "JobDetails": {
      "JobCategory": "string",
      "StudioCode": "string", 
      "Title": "string",
      "ClientCode": "string"
    }
  }
}

Configuration

To add new Celtra-eligible clients, update CELTRA_ELIGIBLE_CLIENTS in the Config class:

CELTRA_ELIGIBLE_CLIENTS = {
    "CIBC", "OLIVER", "ADIDAS", "PAYPAL", 
    "RECKITTBENCKISER", "BAYER", "3M", "RANK",
    "NEW_CLIENT"  # Add here
}

Logging

Activity is logged to both console and json_workflow.log file.

Error Handling

• Invalid JSON files trigger email notifications with file attachment
• Failed file operations are logged with details
• Original files are only removed after successful processing
• Files that fail processing are left in place for manual review