obsidian/01 Projects/ford_qc/Ford QC System.md

name	client	status	tech	local_path	deploy	url	server	tags	created	last_commit	commits	service
Ford QC System	Ford	active	Python Box SDK HTML/Bootstrap JSON ZIP/Pillow systemd	/Users/ai_leed/Documents/Projects/Oliver/ford_qc	sudo systemctl start ford-qc-hotfolder.service	Box folder 332861865120	box-cli	ford qc quality-control box bnp systemd hotfolder	2026-04-14	2026-04-29	23	ford-qc-hotfolder.service

Overview

Ford QC is an automated quality control system that validates Ford Build & Price (BnP) asset packs—ZIP files containing automotive images, metadata, and configuration—against 20 specialized validation checks. It monitors a Box cloud folder for incoming asset packs, processes them through the QC engine, generates detailed HTML reports, and stores results back to Box. The system is production-deployed as a systemd daemon on the box-cli host and handles both MEC (Market Enabled Configuration) and BAU (Business as Usual) asset types.

Tech Stack

• Frontend: HTML/Bootstrap (static report generation)
• Backend: Python 3, Box SDK, Flask-adjacent (CLI + daemon pattern)
• Database: N/A (stateless processing; Box as storage)
• Infrastructure: systemd service, Box cloud integration, Python venv
• AI/ML: N/A
• Key libraries: boxsdk (3.13.0), pillow (11.1.0), python-dotenv (1.0.1), requests (2.32.3)

Architecture

The system comprises three primary execution paths:

1. CLI Entry Point (qc_engine.py): Loads a JSON profile, executes checks in sequence, generates JSON results, and optionally produces HTML reports. Used for manual/ad-hoc testing.

2. QC Module (qc_module.py): Reusable Python module exposing run_qc_checks() and run_qc_profile() functions for programmatic check execution with enhanced error handling.

3. Box Hotfolder Daemon (ford_qc_box_hotfolder_process.py): Production service monitoring Box folder 332861865120 for incoming *_GPAS.zip files. Downloads, processes through QC, uploads JSON/HTML reports to folder 332864939558, and moves processed files to folder 332861653811 with MD5 stamps. Implements file locking, graceful shutdown, and systemd watchdog integration.

Check Modules (13 in checks/ directory):

• zip_filename_check: Enforces _GPAS.zip suffix (Ford 3rd-party requirement)
• unzip_and_verify_check: Extracts ZIP and validates linkingrecord.json presence
• colour_existence_check: Verifies color chip files exist and are accessible
• missing_images_check: Cross-references metadata vs. actual files
• special_requirements_mec_bau: Auto-determines asset type (MEC/BAU) and validates type-specific rules
• mec_powertrain_validation, lifestyle_inventory_validation, image_resolution_check, image_format_check, exterior_interior_pairing_check, base_assets_check, carousel_images_check, linkingrecord_header_validation: Specialized validations per asset type
• html_reporter.py: Converts JSON results to Bootstrap-styled HTML success reports
• html_error_reporter.py: Converts error data to HTML error reports

Data Flow:

Box Input Folder (332861865120)
           ↓
[Hotfolder daemon polls every 60s]
           ↓
Download ZIP to /working_dir
           ↓
Extract + Run 13 QC checks → JSON results
           ↓
Generate HTML reports (success & error)
           ↓
Upload JSON/HTML to Box Report Folder (332864939558)
           ↓
Move processed ZIP to Box Processed Folder (332861653811) + MD5 stamp

Dev Commands

# Install dependencies
pip install -r requirements.txt

# Run QC on a local ZIP file
python qc_engine.py profiles/ford_bnp.json --input_file path/to/asset_pack.zip --reports_dir reports

# Generate HTML report from existing JSON results
python -m checks.html_reporter results.json output_directory/

# Generate HTML error report
python -m checks.html_error_reporter <error_type> <filename> output_directory/

# Test a single check module
python -c "from checks.image_format_check import run_check; print(run_check({'working_dir': 'working'}))"

# Run all tests
python -m pytest tests/

# Start the Box hotfolder daemon (locally)
python ford_qc_box_hotfolder_process.py

# Verify systemd service status (production)
sudo systemctl status ford-qc-hotfolder.service

# View live production logs
journalctl -u ford-qc-hotfolder.service -f

# Restart production service
sudo systemctl restart ford-qc-hotfolder.service

Deployment

• Server: box-cli
• Deploy: sudo systemctl start ford-qc-hotfolder.service (or restart)
• URL: Box folder ID 332861865120 (input), 332864939558 (reports output)
• Port: N/A (daemon; no HTTP port)
• Service: ford-qc-hotfolder.service (systemd)
• Local path: /Users/ai_leed/Documents/Projects/Oliver/ford_qc
• Production path: /home/box-cli/FORD_SCRIPTS/ford_qc_git_prod/ford_qc

Setup:

1. Copy ford-qc-hotfolder.service to /etc/systemd/system/
2. Run sudo systemctl daemon-reload && sudo systemctl enable ford-qc-hotfolder.service
3. Configure .env.prod with Box credentials and folder IDs in the script directory
4. Start: sudo systemctl start ford-qc-hotfolder.service

Environment Variables

All env vars are loaded from .env.prod (production) or .env.dev (dev). Key variables:

• FORD_QC_ENV — Environment name (production or development)
• BOX_SOURCE_FOLDER_ID — Folder monitored for incoming ZIPs (e.g., 332861865120)
• BOX_REPORT_FOLDER_ID — Folder where JSON/HTML reports are uploaded (e.g., 332864939558)
• BOX_PROCESSED_FOLDER_ID — Folder where successfully processed files are moved (e.g., 332861653811)
• BOX_CONFIG_FILE — Path to Box JWT JSON credentials file (default: ford_box_config.json)
• FORD_QC_SCRIPT_DIR — Base directory for QC system (auto-detected if not set)
• DOWNLOAD_PATH — Temp directory for Box downloads (default: download_tmp)
• WORKING_DIR — Directory for ZIP extraction and processing (default: working)
• QC_PROFILE_PATH — QC profile file (default: profiles/ford_bnp.json)
• BOX_CONNECTION_TIMEOUT — Timeout for Box API connection (seconds; default: 30)
• BOX_READ_TIMEOUT — Timeout for Box API read (seconds; default: 90)
• MAX_RETRIES — Max retry attempts for failed operations (default: 3)
• RETRY_BACKOFF_BASE — Base for exponential backoff (seconds; default: 2)
• WATCHDOG_INTERVAL — Systemd watchdog notification interval (seconds; default: 30)
• LOG_LEVEL — Log verbosity (DEBUG, INFO, WARNING, ERROR, CRITICAL; default: INFO)

Known Issues

• No SSH to production without explicit instruction — security constraint
• File locking enforced — do not run multiple hotfolder instances simultaneously
• ZIP filename enforcement — files must end with _GPAS.zip; enforced by zip_filename_check.py
• Box folder IDs hardcoded — in ford_qc_box_hotfolder_process.py; changes require careful review
• Python venv required — production runs from venv/bin/python; system Python not supported
• MEC vs BAU auto-detection — logic in special_requirements_mec_bau.py determines asset type; ensure linkingrecord.json structure is valid
• Ranger ptvl pattern configurable — via profile; see recent git commits for changes
• Exterior/interior pairing validation — uses variant-signature approach; angle 21 support added recently

Git

• Remote: git@bitbucket.org:zlalani/ford_qc.git
• Last 20 commits include: exterior/interior pairing, CV ABM support, linkingrecord header validation, asset count summary, GPAS naming enforcement, carousel image checks, Box config debugging, environment configuration, systemd watchdog improvements, and boxsdk upgrade to 3.13.0

Timeline / Git History

Date	Change
2026-04-29	Add linkingrecord header validation check
2026-04-17	Add asset count summary to QC HTML reports
2026-04-16	Add zip filename check for GPAS naming convention
2026-03-16	Make Ranger ptvl pattern configurable via profile

Sessions

2026-05-01 – Check QC tool configuration and reboot

Asked: Check QC tool configuration and reboot the server to diagnose the issue. Done: Identified that DEV environment has an invalid Box config with incorrect JWT key while PROD config is valid, causing the QC tool failure.

2026-05-01 – Asked | Done | Log

Asked: Asked | Done | Log Done: ---|---|---

2026-05-01 – Can you restart the QC tool

Asked: Can you restart the QC tool and investigate the reported errors? Done: Identified that the DEV service fails with Box API authentication error due to mismatched client credentials between DEV and PROD environments.

2026-05-01 – Check QC tool issues and investigate

Asked: Check QC tool issues and investigate DEV service status on the box-cli server. Done: Connected to box-cli server and reviewed service logs to diagnose QC tool failures.

2026-04-29 – Verify series image counts using SE#

Asked: Verify series image counts using SE# and ABM codes, accounting for SIPS variations. Done: Removed incorrect images and links for ambl2se#r1, verified all 20/20 tests pass, committed, pushed, and deployed to dev.

2026-04-29 – Check series image accounting using SE#

Asked: Check series image accounting using SE# and ABM WERS codes to verify all series are covered. Done: Identified and removed duplicate ambl2se#r1 image linking while retaining series image; restarted ford-qc-hotfolder.service dev environment.

2026-04-29 – Implement a check to validate series

Asked: Implement a check to validate series images using SE# and ABM codes to ensure all series are accounted for. Done: Rewrote check_series_permutations.py with autodetection logic to identify orphan series carousel entries lacking backing images.

2026-04-29 – Check that all series images are

Asked: Check that all series images are accounted for using SE# and ABM WERS codes combination. Done: Implemented series permutation check that detects orphan carousel entries missing backing images and validates SE# and ABM code combinations.

2026-04-29 – Check which commit contains the Ford

Asked: Check which commit contains the Ford image pack naming feature and deploy it to production. Done: Deployed zip filename validation check to production and identified service is running from different path requiring file sync.

2026-04-29 – Asked | Rename image pack files

Asked: Asked | Rename image pack files to end with _GPAS.zip instead of _image.zip to support Ford's 3rd party checker | ford_bnp.json Done: Done | Updated ford_bnp.json to add zip format validation requiring _GPAS.zip suffix | ford_bnp.json

2026-04-29 – Identify which commit introduced the image

Asked: Identify which commit introduced the image pack naming requirement change and apply it to production. Done: Located two files to deploy from commit 16bbd98 — the new zip filename validation check and updated Ford profile configuration.

2026-04-29 – Check which commit contains the image

Asked: Check which commit contains the image pack naming feature and deploy it to production. Done: Identified commit 16bbd98 on origin/main with deployment command ready for the production server.

2026-04-29 – Find the commit implementing the GPAS

Asked: Find the commit implementing the GPAS zip filename naming convention and prepare it for production deployment. Done: Located commit 16bbd98 with the GPAS naming check; clarified deployment path requires server-side git pull and service restart on box-cli-01.

2026-04-29 – Asked | Done | Log

Asked: Asked | Done | Log Done: Add validation checks for linking record top section fields | Implemented format validation for market (3 alphabetic chars), model (dynamic list), year (YYY format), and header fields | LinkingRecord.ts, ValidationRules.ts

2026-04-29 – Add validation checks for linking record

Asked: Add validation checks for linking record data format fields | Implemented format validation for market (3 letters), year (YYY), and model fields with allowlist checking | LinkingRecord.ts, ValidationSchema.ts Done: Pushed code to git and deployed to dev server | Changes committed, pushed to repository, and deployed to development server | deployment scripts executed

2026-04-29 – Add validation checks for linking record

Asked: Add validation checks for linking record top section fields (market, model, year, fullLoad, header). Done: Implemented format validation for market (3-letter codes), model (dynamic list), and year (YYY format) with smoke tests confirming module imports and runs without errors.

2026-04-17 – Where can we find the report

Asked: Where can we find the report and why isn't the previously done task working? Done: Generated report opened in browser using linkingrecord.json and QC results, added Asset Count card showing link and asset counts by section.

2026-04-17 – Asked | Done | Log

Asked: Asked | Done | Log Done: Feature deployment | Pushed changes and restarted dev service | ford-qc-hotfolder.service

2026-04-17 – Where is the report and why

Asked: Where is the report and why isn't the implementation working? Done: Updated implementation to display total linking records and unique assets for overall and per-section views by modifying the call site to pass four values.

2026-04-17 – Asked | Where can the new

Asked: Asked | Where can the new report be found after implementation? | Completed Done: Done | Added two new static methods before _build_head and verified changes | Code updated with new static methods

2026-04-16 – Changed zip file naming convention from

Asked: Changed zip file naming convention from "_image.zip" to "_GPAS.zip" for image pack validation. Done: Updated image pack naming format and added validation check to ensure all packs end with "_GPAS.zip" suffix.

2026-04-16 – Update zip file naming convention from

Asked: Update zip file naming convention from "_image.zip" to "_GPAS.zip" for image pack validation. Done: Implemented zip filename check to enforce "_GPAS.zip" suffix and deployed to production with proper validation messaging.

2026-04-16 – Change image pack naming convention from

Asked: Change image pack naming convention from "_image.zip" to "_GPAS.zip" suffix. Done: Updated naming validation logic and pushed changes to git, resolved merge conflict with .gitignore and .env configuration.

2026-04-16 – Update image pack naming convention from

Asked: Update image pack naming convention from "_image.zip" to "_GPAS.zip" suffix. Done: Modified naming validation logic to enforce "_GPAS.zip" format for image pack files.

2026-04-16 – Change image pack naming convention from

Asked: Change image pack naming convention from "_image.zip" to "_GPAS.zip" suffix. Done: Updated naming validation to enforce _GPAS.zip suffix for all image packs and committed changes to git.

2026-04-16 – Update image pack naming from "_image.zip"

Asked: Update image pack naming from "_image.zip" to "_GPAS.zip" suffix for Ford third-party checker compatibility. Done: Modified image pack validation logic and pushed changes to git, but encountered merge conflict in .gitignore during pull.

2026-04-16 – Update image pack naming from "_image.zip"

Asked: Update image pack naming from "_image.zip" to "_GPAS.zip" suffix requirement. Done: Modified validation logic to enforce _GPAS.zip naming convention and deployed changes via git pull with service restart.

2026-04-16 – Update image pack naming from "_image.zip"

Asked: Update image pack naming from "_image.zip" to "_GPAS.zip" suffix with validation check. Done: Modified naming convention and validation logic, pushed to Git and restarted the ford-qc-hotfolder service.

2026-04-16 – Change image pack naming from "_image.zip"

Asked: Change image pack naming from "_image.zip" to "_GPAS.zip" suffix in the validation check. Done: Updated zip file naming validation logic to require "_GPAS.zip" suffix instead of "_image.zip", and pushed changes to git after resolving merge conflicts.

2026-04-16 – Change image pack naming from "_image.zip"

Asked: Change image pack naming from "_image.zip" to "_GPAS.zip" suffix for Ford QC validation. Done: Updated naming convention and resolved git merge conflict by stashing local changes before pulling updates.

2026-04-16 – Update image pack naming convention from

Asked: Update image pack naming convention from _image.zip to _GPAS.zip | Changed zip file naming pattern validation and .gitignore file | .gitignore, image_pack_validator.py Done: —

2026-04-16 – Update image pack naming validation to

Asked: Update image pack naming validation to require "_GPAS.zip" suffix instead of "_image.zip" Done: Created zip filename check module and integrated it into Ford QC validation pipeline

2026-04-16 – Update zip file naming requirement from

Asked: Update zip file naming requirement from "_image.zip" to "_GPAS.zip" suffix. Done: Created filename validation check, integrated into Ford BNP profile, and verified all test scenarios pass.

2026-04-16 – Update image pack naming convention to

Asked: Update image pack naming convention to use "_GPAS.zip" suffix instead of "_image.zip". Done: Modified naming validation logic and updated all image pack references to enforce the new "_GPAS.zip" suffix format.

2026-04-14 – Project catalogued

Done: Added to Obsidian second brain with full details.

---

Change Log

Date	Requested	Changed	Files
2026-05-01	QC tool config fix	Restore valid JWT key in ford_box_config.json, align DEV config with PROD	ford_box_config.json, .env
2026-05-01	Check QC tool server issues and attempt reboot	Verified Box folder configurations for DEV environment and checked PROD .env folder ID settings	Server config, .env
2026-05-01	QC tool debugging	Box API 400 error, clientID/publicKeyID mismatch between DEV and PROD	ford-qc-hotfolder.service config
2026-05-01	QC tool investigation	Server connection established, service logs reviewed	box-cli server logs
2026-04-29	Obsidian project docs enrichment	Added session tracking, deployment details, stack info, URLs, and change history to project notes	project-notes.ts, obsidian-sync.ts, project-enrichment.ts
2026-04-29	Series image verification	Removed images/links for ambl2se#r1, verified test results, committed and deployed	Image assets, test configuration, deployment config
2026-04-29	Series image validation	Removed duplicate ambl2se#r1 linking, restarted service	Image linking config, ford-qc-hotfolder.service
2026-04-29	Series image validation	Added _autodetect_mode() function, orphan detection logic	checks/check_series_permutations.py
2026-04-29	Series validation check	Added orphan detection for carousel entries, implemented SE#/ABM mode autodetection	checks/check_series_permutations.py
2026-04-29	Image pack naming validation	Added zip_filename_check to Ford profile, copied check script to prod	checks/zip_filename_check.py, profiles/ford_bnp.json
2026-04-29	Log	Image pack naming convention	Added _GPAS.zip validation check, reordered validation rules
2026-04-29	Image pack naming update	Added _GPAS.zip validation check, Updated Ford profile with new validation rule	checks/zip_filename_check.py, profiles/ford_bnp.json
2026-04-29	Image pack naming	Updated naming format from _image.zip to _GPAS.zip, added validation check	FORD_ASSET_PACK_QC_DEV service
2026-04-29	GPAS filename validation	Added _GPAS.zip naming check, removed _image.zip legacy support	Image pack validator, deployment config
2026-04-29	Pushed code to Git and deployed to dev server	Committed 3 files to main branch (5abd481) and restarted ford-qc-hotfolder service on dev with successful git pull	ford_qc_git_dev/ford_qc, service config
2026-04-29	Linking record validation	market length/alphabet check, year format check, model allowlist validation	LinkingRecord.ts, ValidationSchema.ts
2026-04-29	Validation checks	Added market/model/year format validation, removed obsolete function	unzip_and_verify_check.py, linking_record_validator.py
2026-04-17	Report generation and deployment	Generated report with Asset Count card, linked QC results to summary	linkingrecord.json, test_reports, box-cli dev deployment
2026-04-17	Report location	Code pushed to git and deployed to box-cli dev folder	dev deployment
2026-04-17	Report display implementation	Import validation, function signature updated to accept four parameters	call site, implementation file
2026-04-17	New static methods	Added two static methods before _build_head, verified implementation	Source file
2026-04-16	Image pack naming	Zip suffix changed from _image to _GPAS, validation check added	.gitignore, validation script
2026-04-16	Zip naming validation	zip_filename_check implementation, validation logic, error messaging	.gitignore, validation module files
2026-04-16	Zip naming update	Validation logic for _GPAS.zip suffix, .env configuration setup	.gitignore, utils/config.py, .env.example
2026-04-16	Zip naming convention	Filename suffix validation logic, file pattern matching	validation.py, config.py
2026-04-16	Zip naming convention	Updated suffix validation from _image.zip to _GPAS.zip	.gitignore, image pack validator
2026-04-16	Image pack naming	Suffix pattern changed from _image.zip to _GPAS.zip, validation logic updated	.gitignore, validation module files
2026-04-16	Image pack naming validation	Suffix check from _image.zip to _GPAS.zip	.gitignore, ford_qc validation module
2026-04-16	Image pack naming	Zip suffix changed from _image.zip to _GPAS.zip, validation check added	.gitignore, ford_qc service configuration
2026-04-16	Zip naming validation	Suffix changed from _image.zip to _GPAS.zip	Image pack validator, .gitignore, ford_box_config.json
2026-04-16	Zip naming convention	Suffix changed from _image.zip to _GPAS.zip	.gitignore, ford_box_config.json
2026-04-16	zip naming validation	New check module creation, profile integration, documentation	checks/zip_filename_check.py, profiles/ford_bnp.json, CLAUDE.md
2026-04-16	Zip naming validation	New check module creation, profile integration, test validation	checks/zip_filename_check.py, profiles/ford_bnp.json, CLAUDE.md
2026-04-16	Image pack naming	Zip suffix validation, pack name references	validation.ts, config.ts, constants.ts
2026-03-16	Configurable pattern	Ranger ptvl pattern via profile JSON	qc_engine.py

Related

• ford-gechub-sftp/Ford SFTP Transfer