Ford Asset Pack SFTP Transfer System

Automated system for transferring Ford asset pack ZIP files from Box cloud storage to Ford's GECHUB SFTP servers, with archival and email notifications.

Overview

This system monitors three Box folders (PROD, EDU, QA) for new Ford asset pack ZIP files. When files are detected, it:

1. Downloads files from Box
2. Uploads to appropriate GECHUB SFTP environment
3. Archives files in Box
4. Sends email notifications via Mailgun

The system supports both one-shot (manual) and daemon (continuous polling) operation modes with robust error handling, automatic retries, and comprehensive logging.

Features

Core Functionality

• ✅ Multi-Environment Support - Separate credentials for PROD, EDU, QA environments
• ✅ Dual-Factor SFTP Authentication - Public key + keyboard-interactive password
• ✅ Box Integration - JWT authentication, automatic archiving
• ✅ Email Notifications - Success and error alerts via Mailgun

Reliability & Error Handling

• ✅ Automatic Retries - Exponential backoff for failed operations
• ✅ Per-File Isolation - One file's failure doesn't stop others
• ✅ Service Outage Tolerance - Continues running if services temporarily unavailable
• ✅ Rate-Limited Alerts - Prevents email spam during outages
• ✅ Graceful Shutdown - Completes current operation before stopping

Daemon Mode

• ✅ Continuous Monitoring - Polls Box folders every 5 minutes (configurable)
• ✅ Systemd Integration - Auto-restart, boot-on-start, proper logging
• ✅ Signal Handling - Responds to SIGTERM/SIGINT gracefully
• ✅ Configurable Timeouts - Separate timeouts for Box, SFTP, Mailgun

Architecture

┌─────────────┐
│  Box Folders │
│ PROD/EDU/QA │
└──────┬──────┘
       │ Download ZIP
       ▼
┌─────────────────┐         ┌──────────────┐
│   Temp Storage  │────────▶│ GECHUB SFTP  │
│  (Temporary)    │  Upload │ PROD/EDU/QA  │
└─────────────────┘         └──────────────┘
       │
       │ Archive
       ▼
┌─────────────────┐         ┌──────────────┐
│  Box Archive    │         │   Mailgun    │
│ prodgpas/edu/qa │◀────────│ Notifications│
└─────────────────┘         └──────────────┘

Authentication Flow

Box: JWT authentication via ford_box_config.json

SFTP: Dual-factor authentication

1. Public key (from .env PRIVATE_KEY_PATH)
2. Keyboard-interactive password (from .env PASSWORD)

Mailgun: API key authentication

Quick Start

Prerequisites

• Python 3.7+
• Linux system with systemd (for daemon mode)
• Access to:
  • Box account with JWT credentials
  • Ford GECHUB SFTP servers
  • Mailgun account

Installation

# Clone or navigate to project directory
cd /path/to/Ford_box_sFTP_mover

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure credentials
cp .env.example .env
nano .env  # Fill in your credentials

# Verify Box JWT config exists
ls ford_box_config.json

# Set SSH key permissions
chmod 600 ./prod_key ./qa_key ./edu_key

Test Installation

# Test SFTP connection (single environment)
python test_sftp_upload.py --env qa --file test_upload.txt

# Test all environments
python test_sftp_upload.py --all --file test_upload.txt

# Run script once (manual mode)
python ford_asset_pack_sftp.py --once

Configuration

Environment Variables (.env)

# =============================================================================
# SFTP Credentials (per environment)
# =============================================================================
PROD_SERVER_URL=sftpmft.gechub.ford.com
PROD_USERNAME=your_username
PROD_PASSWORD=your_password
PROD_PRIVATE_KEY_PATH=/path/to/prod_key
PROD_DESTINATION_PATH=/destination/path

QA_SERVER_URL=sftpmftqa.gechub.ford.com
QA_USERNAME=your_username
QA_PASSWORD=your_password
QA_PRIVATE_KEY_PATH=/path/to/qa_key
QA_DESTINATION_PATH=/destination/path

EDU_SERVER_URL=sftpmft.gechub.ford.com
EDU_USERNAME=your_username
EDU_PASSWORD=your_password
EDU_PRIVATE_KEY_PATH=/path/to/edu_key
EDU_DESTINATION_PATH=/destination/path

# =============================================================================
# Daemon Configuration
# =============================================================================
POLL_INTERVAL_SECONDS=300           # How often to check Box (5 minutes)
MAX_RETRIES=3                       # Retry attempts per operation
RETRY_INITIAL_DELAY_SECONDS=5       # First retry delay
RETRY_MAX_DELAY_SECONDS=300         # Max delay between retries
RETRY_BACKOFF_MULTIPLIER=2.0        # Exponential backoff multiplier

# =============================================================================
# Timeout Configuration
# =============================================================================
BOX_API_TIMEOUT_SECONDS=120         # Box operations timeout
SFTP_UPLOAD_TIMEOUT_SECONDS=1200    # SFTP upload timeout (20 min)
MAILGUN_TIMEOUT_SECONDS=30          # Email notification timeout

Box Folder Mapping

Edit in ford_asset_pack_sftp.py if needed:

FOLDER_TO_ENV = {
    '158875001805': 'prod',   # PROD Box folder → PROD SFTP
    '158874882387': 'edu',    # EDU Box folder → EDU SFTP
    '158875650409': 'qa'      # QA Box folder → QA SFTP
}

Usage

One-Shot Mode (Manual Execution)

Process files once and exit:

python ford_asset_pack_sftp.py --once

Use cases:

• Manual file processing
• Testing after configuration changes
• Scheduled via cron (not recommended, use daemon mode instead)

Daemon Mode (Continuous Operation)

Foreground (Testing)

# Run in foreground to see logs
python ford_asset_pack_sftp.py --daemon

# Press Ctrl+C to stop

Background (Systemd Service)

# Install service
sudo ./install_service.sh

# Start daemon
sudo systemctl start ford-asset-pack-sftp

# Check status
sudo systemctl status ford-asset-pack-sftp

# Enable on boot
sudo systemctl enable ford-asset-pack-sftp

# View live logs
sudo journalctl -u ford-asset-pack-sftp -f

# Stop daemon
sudo systemctl stop ford-asset-pack-sftp

# Restart daemon
sudo systemctl restart ford-asset-pack-sftp

Testing

Test SFTP Connections

Before running the full script, test SFTP connectivity:

# Test single environment
python test_sftp_upload.py --env qa --file test_upload.txt

# Test all environments
python test_sftp_upload.py --all --file test_upload.txt

# Test with real file
python test_sftp_upload.py --env prod --file sample_pack.zip

Manual SFTP Test

# Test connection manually
sftp -i ./qa_key username@sftpmftqa.gechub.ford.com

# At sftp> prompt:
sftp> pwd
sftp> ls
sftp> put test_file.txt
sftp> bye

Test Box Access

# Test in Python
python3
>>> from boxsdk import JWTAuth, Client
>>> auth = JWTAuth.from_settings_file('ford_box_config.json')
>>> client = Client(auth)
>>> folder = client.folder('158875650409').get()
>>> print(folder.name)
>>> exit()

Monitoring & Logs

View Logs

# Live logs (follow mode)
sudo journalctl -u ford-asset-pack-sftp -f

# Last 100 lines
sudo journalctl -u ford-asset-pack-sftp -n 100

# Today's logs
sudo journalctl -u ford-asset-pack-sftp --since today

# Search for errors
sudo journalctl -u ford-asset-pack-sftp | grep ERROR

# Export to file
sudo journalctl -u ford-asset-pack-sftp --since today > logs_$(date +%Y%m%d).log

Log Messages

Key log patterns:

Starting processing cycle          # Cycle beginning
Found zip: filename.zip            # File detected
✅ Successfully processed          # File completed
⚠️  REMINDER: Delete test file     # Test file uploaded
Waiting 300 seconds               # Polling interval
Daemon shutdown complete          # Graceful stop

Health Check

# Is daemon running?
systemctl is-active ford-asset-pack-sftp

# When did it last log?
sudo journalctl -u ford-asset-pack-sftp -n 1

# Any errors in last hour?
sudo journalctl -u ford-asset-pack-sftp --since "1 hour ago" | grep -i error

Troubleshooting

Common Issues

Service Won't Start

# Check status
sudo systemctl status ford-asset-pack-sftp

# View recent errors
sudo journalctl -u ford-asset-pack-sftp -n 50

# Test manually
python ford_asset_pack_sftp.py --once

Common causes:

• Missing .env file or variables
• Invalid Box JWT config
• Wrong Python path in service file
• Missing dependencies

Files Not Processing

Check:

1. Files exist in Box folders
2. Files are ZIP format
3. Daemon is running: sudo systemctl is-active ford-asset-pack-sftp
4. No errors in logs: sudo journalctl -u ford-asset-pack-sftp -p err

SFTP Upload Failures

# Check SFTP logs
sudo journalctl -u ford-asset-pack-sftp | grep "SFTP"

# Test connection manually
sftp -i ./prod_key username@sftpmft.gechub.ford.com

# Verify credentials in .env
grep PROD_USERNAME .env
grep PROD_SERVER_URL .env

Common causes:

• Wrong password or username
• SSH key permissions (must be 600)
• Network connectivity issues
• Destination path doesn't exist

Authentication Failures

# Box authentication
grep "box" -i $(sudo journalctl -u ford-asset-pack-sftp --since today | grep -i error)

# SFTP authentication
grep "Authentication failed" $(sudo journalctl -u ford-asset-pack-sftp --since today)

Solutions:

• Verify ford_box_config.json is valid
• Check SFTP credentials in .env
• Ensure SSH keys have correct permissions: chmod 600 ./*_key
• Test manual connections

Getting Help

1. Check documentation:

   • TESTING_GUIDE.md - Comprehensive testing procedures
   • DAEMON_MODE.md - Daemon operation details
   • MIGRATION_NOTES.md - Migration from legacy system

2. Review logs:

   sudo journalctl -u ford-asset-pack-sftp -n 200 > debug.log
   

3. Test components individually:

   • Test SFTP: test_sftp_upload.py
   • Test Box: Python interactive shell
   • Test one-shot: python ford_asset_pack_sftp.py --once

Project Structure

Ford_box_sFTP_mover/
├── ford_asset_pack_sftp.py           # Main application
├── ford_box_config.json               # Box JWT credentials (not in git)
├── test_sftp_upload.py                # SFTP connection test script
├── test_upload.txt                    # Sample test file
│
├── .env                               # Environment configuration (not in git)
├── .env.example                       # Configuration template
├── .gitignore                         # Git ignore rules
│
├── requirements.txt                   # Python dependencies
├── venv/                              # Python virtual environment
│
├── ford-asset-pack-sftp.service       # Systemd service file
├── install_service.sh                 # Service installation script
├── uninstall_service.sh               # Service removal script
│
├── prod_key                           # PROD SSH private key (not in git)
├── qa_key                             # QA SSH private key (not in git)
├── edu_key                            # EDU SSH private key (not in git)
│
├── README.md                          # This file
├── DAEMON_MODE.md                     # Daemon operation guide
├── TESTING_GUIDE.md                   # Testing procedures
├── MIGRATION_NOTES.md                 # Migration documentation
├── CLAUDE.md                          # Project context for Claude Code
│
└── EXAMPLE_sftp_transfer_GECHUB.py    # POC script (reference only)

File Lifecycle

1. Detection: ZIP file appears in Box folder (PROD/EDU/QA)
2. Download: File downloaded to temporary directory
3. Upload: File uploaded to corresponding GECHUB SFTP environment
4. Archive: File moved to Box archive subfolder (prodgpas/edugpas/qagpas)
5. Notification: Success email sent via Mailgun
6. Cleanup: Temporary files automatically deleted

Error handling:

• If upload fails after retries: File stays in Box folder, error notification sent
• If Box move fails: File uploaded but remains in original folder
• Daemon continues processing other files regardless of individual failures

Dependencies

Python Packages

boxsdk          # Box API integration
requests        # HTTP client for Mailgun
pexpect         # Interactive process control (SFTP)
python-dotenv   # Environment variable management

System Requirements

• Python 3.7 or higher
• Linux with systemd (for daemon mode)
• OpenSSH sftp command-line client
• Network access to:
  • Box API (api.box.com)
  • GECHUB SFTP servers
  • Mailgun API (api.mailgun.net)

Security Considerations

Credentials

• ✅ .env file is in .gitignore (never commit credentials)
• ✅ SSH private keys should have 600 permissions
• ✅ Box JWT credentials stored separately
• ✅ Mailgun API key in environment variables

Access Control

• Run daemon as dedicated user (not root)
• Limit file permissions on sensitive files:

  chmod 600 .env
  chmod 600 *_key
  chmod 600 ford_box_config.json
  

Network Security

• SFTP connections use SSH encryption
• Box API uses HTTPS
• Mailgun API uses HTTPS

Maintenance

Regular Tasks

Daily:

• Monitor logs for errors: sudo journalctl -u ford-asset-pack-sftp -p err --since today
• Verify daemon is running: systemctl is-active ford-asset-pack-sftp

Weekly:

• Check Box archive folders for proper organization
• Review error notification emails
• Clean up test files from SFTP servers

Monthly:

• Review and rotate logs if needed
• Update Python dependencies: pip install --upgrade -r requirements.txt
• Verify SFTP credentials haven't expired

Updates

# Stop daemon
sudo systemctl stop ford-asset-pack-sftp

# Update code
git pull  # or copy new files

# Update dependencies
source venv/bin/activate
pip install --upgrade -r requirements.txt

# Test changes
python ford_asset_pack_sftp.py --once

# Restart daemon
sudo systemctl start ford-asset-pack-sftp

Performance

Typical Processing Times

• Small files (<10MB): ~5-15 seconds per file
• Medium files (10-100MB): ~15-60 seconds per file
• Large files (100MB-1GB): ~1-10 minutes per file

Polling Overhead

• CPU: Minimal (mostly idle between polls)
• Memory: ~50-100MB resident
• Network: Bandwidth only during file transfers

Scaling

Current configuration handles:

• Up to 1000 files per Box folder
• Concurrent processing of 3 environments
• Files up to 2GB (limited by SFTP timeout)

To process more:

• Increase SFTP_UPLOAD_TIMEOUT_SECONDS for larger files
• Decrease POLL_INTERVAL_SECONDS for more frequent checks
• Run multiple instances for different Box folders

License

Proprietary - Oliver Agency internal use only.

Authors

• Michael Clervi - Development & Implementation
• Oliver Agency - Product Owner

Acknowledgments

• Ford Motor Company for GECHUB infrastructure
• Box.com for cloud storage API
• Mailgun for email notifications

Support

For issues or questions:

1. Check documentation in this repository
2. Review logs: sudo journalctl -u ford-asset-pack-sftp
3. Test components individually using test scripts
4. Contact: simonharnden@oliver.agency

Changelog

v2.0.0 (2025-10-23)

• ✨ Migrated from legacy paramiko to OpenSSH sftp
• ✨ Added daemon mode with continuous polling
• ✨ Implemented retry logic with exponential backoff
• ✨ Added systemd service integration
• ✨ Enhanced error handling and logging
• ✨ Added rate-limited error notifications
• ✨ Configurable timeouts and retry settings
• ✨ Comprehensive documentation and testing tools

v1.0.0 (Legacy)

• Initial implementation with paramiko
• Single-factor SSH key authentication
• Manual execution only
• Basic error handling