michael d7049b5de2 readme and claude.md updates

2025-09-03 07:47:09 -05:00

14 KiB

Raw Permalink Blame History

Ford BnP Asset Pack Quality Control System

A comprehensive quality control (QC) system for Ford Build and Price (BnP) asset packs, featuring automated processing, extensive validation checks, and integrated Box cloud storage workflow.

Overview
Quick Start
Architecture
Installation
Usage
QC Checks
Development
Production Deployment
Troubleshooting
Contributing

Overview

The Ford BnP QC system validates asset packs containing automotive images, metadata, and configuration files to ensure they meet Ford's quality standards before deployment to customer-facing applications.

Key Features

13 Specialized QC Checks: Comprehensive validation of images, metadata, and business rules
Automated Box Integration: Monitors cloud folders, processes files, and delivers reports automatically
Rich HTML Reports: Bootstrap-styled reports with detailed issue breakdowns and fix instructions
Error Recovery: Comprehensive error handling with actionable user guidance
Production Ready: Systemd service with logging, file locking, and graceful shutdown
Extensible Architecture: Modular design for easy addition of new validation checks

Supported Asset Types

MEC (Market Enabled Configuration): Complex multi-variant asset packs
BAU (Business as Usual): Standard single-variant asset packs
Color Chips: Paint color sample images
Lifestyle Images: Marketing and lifestyle photography
Layer-based Images: Multi-layer PNG images with transparency

Quick Start

Command Line Usage

# Run QC checks on a local asset pack
python qc_engine.py profiles/ford_bnp.json --input_file path/to/asset_pack.zip --reports_dir reports

# Generate HTML report from existing JSON results
python -m checks.html_reporter results.json output_directory/

# Test a specific check module
python -c "from checks.image_format_check import run_check; print(run_check({'working_dir': 'working'}))"

Production Service

# Start the Box hotfolder monitoring service
sudo systemctl start ford-qc-hotfolder.service

# Check service status
sudo systemctl status ford-qc-hotfolder.service

# View live logs
sudo journalctl -u ford-qc-hotfolder.service -f

Architecture

System Components

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│   Box Cloud     │    │  QC Engine   │    │ HTML Reporter   │
│                 │    │              │    │                 │
│ • Source Folder │◄──►│ • Profile    │───►│ • Success Report│
│ • Report Folder │    │   Loader     │    │ • Error Report  │
│ • Processed     │    │ • Check      │    │ • Bootstrap UI  │
│   Folder        │    │   Executor   │    │                 │
└─────────────────┘    └──────────────┘    └─────────────────┘
         │                       │                       
         │              ┌──────────────┐                
         │              │ 13 QC Checks │                
         │              │              │                
         └──────────────│ • Images     │                
                        │ • Metadata   │                
                        │ • Business   │                
                        │   Rules      │                
                        └──────────────┘

Data Flow

Input: Asset pack ZIP files uploaded to Box or provided via command line
Processing: Extraction → Validation → Result aggregation
Output: HTML reports with pass/fail status and detailed findings
Distribution: Reports delivered via Box cloud storage or local filesystem

Installation

Prerequisites

Python 3.8 or higher
Virtual environment (recommended)
Box Developer Account (for production integration)

Local Development Setup

# Clone the repository
git clone <repository-url>
cd Ford_BnP_QC

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Create necessary directories
mkdir -p working reports logs

Production Setup

See SERVICE_SETUP.md for complete production deployment instructions including systemd service configuration and Box API setup.

Usage

Profile Configuration

QC profiles define which checks to run and their configurations. The main profile profiles/ford_bnp.json includes all production checks:

[
  {
    "script": "checks.unzip_and_verify_check",
    "config": {
      "expected_file": "linkingrecord.json",
      "working_dir": "__WORKING_DIR__"
    }
  },
  {
    "script": "checks.colour_existence_check", 
    "config": {
      "working_dir": "__WORKING_DIR__",
      "linkingrecord_filename": "linkingrecord.json"
    }
  }
]

Running QC Checks

Local Asset Pack:

python qc_engine.py profiles/ford_bnp.json --input_file asset_pack.zip

Custom Profile:

python qc_engine.py profiles/custom_profile.json --input_file asset_pack.zip --reports_dir custom_reports

Programmatic Usage:

import qc_module

# Run checks and get JSON results
results = qc_module.run_qc_checks('profiles/ford_bnp.json', 'asset_pack.zip')
print(results)

Report Generation

The system automatically generates two types of reports:

Success Reports: Detailed breakdown of all checks with pass/fail status Error Reports: User-friendly error messages with actionable fix instructions

Both report types use Bootstrap styling for professional appearance and mobile compatibility.

QC Checks

Core Validation Checks

Check Name	Purpose	Key Validations
`unzip_and_verify_check`	Archive extraction	ZIP integrity, required files present
`colour_existence_check`	Color validation	Color chip files accessible
`missing_images_check`	Asset completeness	All referenced images present
`image_format_check`	Format compliance	PNG/JPG/AVIF format requirements
`image_resolution_check`	Quality standards	Minimum resolution requirements
`file_size_check`	Performance optimization	Maximum file size limits
`layer_depth_check`	Image complexity	Layer depth analysis
`image_linking_check`	Metadata integrity	Image-metadata relationships

Business Rule Checks

Check Name	Purpose	Key Validations
`special_requirements_mec_bau`	Configuration type	MEC vs BAU requirement compliance
`mec_powertrain_validation`	MEC-specific rules	Powertrain configuration validation
`lifestyle_inventory_validation`	Inventory consistency	Lifestyle asset inventory validation
`check_series_permutations`	Variant validation	Series permutation compliance
`business_data_check`	Data integrity	Business data structure validation

Check Development

Each check module implements a standardized interface:

def run_check(config):
    """
    Execute QC validation logic.
    
    Args:
        config (dict): Configuration parameters including working_dir
        
    Returns:
        dict: Result with status and details
            - status: "passed" | "failed" | "error"
            - details: Additional information and failure specifics
    """
    # Check implementation
    return {"status": "passed", "details": "All validations passed"}

Development

Project Structure

Ford_BnP_QC/
├── checks/                 # QC check modules and reporting
│   ├── *.py               # Individual check implementations
│   ├── html_reporter.py   # Success report generator
│   └── html_error_reporter.py  # Error report generator
├── utils/                  # Utility modules
│   ├── path_resolver.py   # Working directory management
│   └── check_helpers.py   # Common check utilities
├── profiles/              # QC configuration files
│   └── ford_bnp.json     # Production check profile
├── tests/                 # Test files and validation scripts
├── example packs/         # Sample asset packs (37GB+)
├── qc_engine.py          # Main CLI script
├── qc_module.py          # Reusable QC functions
└── ford_qc_box_hotfolder_process.py  # Production service

Adding New Checks

Create Check Module:
```
touch checks/new_validation_check.py
```

Implement Check Interface:

def run_check(config):
    working_dir = config.get("working_dir", "working")
    # Implement validation logic
    return {"status": "passed"}

Add to Profile:

{
  "script": "checks.new_validation_check",
  "config": {
    "working_dir": "__WORKING_DIR__",
    "custom_param": "value"
  }
}

Test Implementation:

python qc_engine.py test_profile.json --input_file sample.zip

Testing

Unit Testing:

# Test individual checks
python -c "from checks.image_format_check import run_check; print(run_check({'working_dir': 'working'}))"

# Run test suite
python -m pytest tests/

Integration Testing:

# Test complete workflow with sample data
python qc_engine.py profiles/ford_bnp.json --input_file "example packs/sample_pack.zip"

Error Report Testing:

# Generate sample error reports
python -m checks.html_error_reporter zip_extraction corrupted.zip reports/
python -m checks.html_error_reporter json_parsing invalid.zip reports/

Production Deployment

Box Integration

The production system integrates with three Box folders:

Source Folder (332861865120): Monitors for new asset pack uploads
Report Folder (332864939558): Delivers QC reports to users
Processed Folder (332861653811): Archives successful files with MD5 stamps

Service Configuration

The system runs as a systemd service with the following features:

Automatic Restart: Service restarts on failure with exponential backoff
File Locking: Prevents concurrent processing with filesystem locks
Graceful Shutdown: Handles SIGTERM/SIGINT for clean service stops
Comprehensive Logging: Rotating logs with configurable verbosity

Monitoring

Log Files:

Service logs: /home/box-cli/FORD_SCRIPTS/FORD_ASSET_PACK_QC_DEV/log/ford_qc_script.log
System logs: journalctl -u ford-qc-hotfolder.service

Health Checks:

# Check service status
sudo systemctl status ford-qc-hotfolder.service

# Monitor processing activity
tail -f /home/box-cli/FORD_SCRIPTS/FORD_ASSET_PACK_QC_DEV/log/ford_qc_script.log

# Check for lock files (indicates active processing)
ls -la /tmp/ford_qc_script.lock

Troubleshooting

Common Issues

Service Won't Start:

# Check permissions
sudo chown -R box-cli:box-cli /home/box-cli/FORD_SCRIPTS/FORD_ASSET_PACK_QC_DEV

# Verify Python environment
source /home/box-cli/FORD_SCRIPTS/FORD_ASSET_PACK_QC_DEV/venv/bin/activate
python -c "import boxsdk; print('Box SDK available')"

Box Authentication Failures:

# Verify Box config file exists and is valid
cat ford_box_config.json | python -m json.tool

# Test Box connection
python -c "from boxsdk import JWTAuth, Client; auth = JWTAuth.from_settings_file('ford_box_config.json'); client = Client(auth); print('Connected')"

Processing Errors:

# Check working directory permissions
ls -la working/

# Verify example pack integrity  
unzip -t "example packs/sample_pack.zip"

# Test individual checks
python -c "from checks.unzip_and_verify_check import run_check; print(run_check({'input_file': 'sample.zip', 'working_dir': 'working'}))"

Error Report Types

The system generates specific error reports for different failure scenarios:

zip_extraction: ZIP corruption, password protection, format issues
json_parsing: linkingrecord.json syntax errors, encoding problems
file_access: Permission issues, missing files, disk space problems
qc_profile: Invalid profile configuration, missing check modules
check_execution: Runtime errors in individual QC checks
network_upload: Box connectivity, authentication, upload failures

Support Resources

Service Logs: Primary source for diagnosing issues
Error Reports: User-friendly HTML reports with fix instructions
Test Data: Extensive example packs for validation and debugging
Development Tools: Individual check testing and profile validation

Contributing

Development Workflow

Fork Repository: Create personal fork for development
Feature Branch: Create branch for new features or fixes
Implementation: Follow existing code patterns and documentation standards
Testing: Validate with example packs and unit tests
Pull Request: Submit with comprehensive description and test results

Code Standards

Python Style: Follow PEP 8 with 4-space indentation
Type Hints: Use typing annotations for function parameters and returns
Documentation: Google-style docstrings with parameter and return descriptions
Error Handling: Comprehensive try/catch with specific exception types
Logging: Structured logging with appropriate verbosity levels

Review Process

All changes require review focusing on:

Functionality: Correct implementation of requirements
Quality: Code style, documentation, and test coverage
Integration: Compatibility with existing checks and workflows
Security: Safe file handling and input validation
Performance: Efficient processing of large asset packs

License

Contact

For questions, support, or contributions, please contact the Ford Digital Asset Management team.

14 KiB Raw Permalink Blame History