veo3/video-generation-lifecycle.md

Video Generation Lifecycle Documentation

Overview

The Veo Video Generation System implements a comprehensive job lifecycle with queue management, real-time status updates, and complete file cleanup.

Lifecycle States

1. Job Submission

• Frontend: User submits video generation request (1-4 videos per request)
• Backend: Creates unique job_id, validates parameters, creates job folder
• Status: queued
• Location: Added to global job queue with FIFO ordering

2. Queue Management

• Processing Limit: Maximum 2 concurrent jobs
• Queue Size: Unlimited per user
• Ordering: First In, First Out (FIFO)
• UI Display: Three sections (Processing, Queued, History)

3. Job Processing

• Status Progression: queued → starting → uploading_image → generating → processing → downloading
• Google API Integration: Multiple API calls for requested video count
• Progress Updates: Real-time polling every 2 seconds
• File Management: Temporary local storage and GCS integration

4. Completion States

Success Path

• Status: completed
• Actions Available: Download buttons
• Auto-cleanup: Job automatically deleted 5 seconds after download
• Files: All temporary files cleaned up

Failure Path

• Status: failed or cancelled
• Actions Available: Retry, Delete buttons
• Location: Moved to History section
• Recovery: Retry re-queues job, Delete removes completely

5. User Actions

Cancel

• Queued Jobs: Removed from queue, moved to History
• Processing Jobs: Stops processing, frees up processing slot
• Result: Status changes to cancelled

Retry

• Applicable: Failed or cancelled jobs only
• Action: Re-queues job with original parameters
• Status: Returns to queued

Delete

• Complete Cleanup:
  • Removes job from memory tracking
  • Deletes local job folder (temp_downloads/job_{job_id}/)
  • Removes GCS files (images, videos)
  • Frees processing slots if applicable
• Result: Job disappears completely from UI

6. File Management

Local Files

temp_downloads/
├── job_{job_id}/
│   ├── uploaded_image.jpg
│   ├── generated_video_1.mp4
│   ├── generated_video_2.mp4
│   └── all_videos.zip

GCS Files

• Temporary images: temp_images/{job_id}_{timestamp}.jpg
• Generated videos: Auto-managed by Google Veo API
• Cleanup: Automatic on job deletion

7. Timeout Handling

• Google API: Indefinite polling (30-second intervals)
• Apache Proxy: 600 seconds (10 minutes)
• GCS Operations: 300 seconds (5 minutes)
• Auto-retry: Up to 3 attempts with exponential backoff

Job Queue Architecture

Backend Components

• Job Status Tracking: In-memory dictionary with job details
• Processing Queue: Array of job IDs being processed
• User Job Counts: Per-user job tracking for management
• Concurrent Processing: Maximum 2 jobs simultaneously

Frontend Components

• QueueManager: Real-time job display with status updates
• VideoForm: Job submission with unlimited queue support
• API Service: RESTful endpoints for job operations

API Endpoints

Job Operations

• POST /api/generate - Submit new job
• GET /api/status/{job_id} - Get job status
• POST /api/cancel/{job_id} - Cancel job
• POST /api/retry/{job_id} - Retry failed job
• DELETE /api/delete/{job_id} - Delete job completely

Downloads

• GET /api/download/{job_id} - Download generated content
• GET /api/download/{job_id}/video/{index} - Individual video download

Error Handling

• Network Timeouts: Automatic retry with longer timeouts
• API Failures: Detailed error messages with retry options
• File Cleanup: Guaranteed cleanup even on failures
• User Feedback: Real-time error display with recovery actions

Queue Display Sections

1. Currently Processing (Blue Highlight)

• Shows jobs actively generating videos
• Maximum 2 jobs displayed simultaneously
• Real-time progress bars with percentage completion
• Available actions: Cancel, Delete
• Automatic transition to History when complete

2. In Queue (Orange Highlight)

• Shows jobs waiting for available processing slots
• Ordered by submission time (FIFO)
• Queue position indicator (e.g., "Queue Position: 3")
• Available actions: Cancel, Delete
• Automatic promotion to Processing when slot opens

3. History (Standard Styling)

• Shows completed, failed, or cancelled jobs
• Ordered by completion time (most recent first)
• Different actions based on final status:
  • Completed: Download All, Individual Video Downloads
  • Failed: Retry, Delete
  • Cancelled: Retry, Delete

Job Status Transitions

Submit → queued → starting → uploading_image → generating → processing → downloading → completed
           ↓         ↓           ↓               ↓             ↓             ↓
        cancel   cancel      cancel          cancel        cancel        auto-delete
           ↓         ↓           ↓               ↓             ↓        (after download)
      cancelled → retry → queued (restart cycle)
                    ↓
                 delete → removed completely

Performance Characteristics

Concurrent Processing

• Global Limit: 2 jobs processing simultaneously
• Queue Length: Unlimited (no per-user restrictions)
• Processing Time: 30-300 seconds per job depending on complexity
• Queue Wait Time: Depends on queue length and processing times

File Sizes and Storage

• Input Images: Up to 10MB, auto-converted to JPEG
• Output Videos: Typically 5-50MB per video
• Temporary Storage: Local and GCS, cleaned automatically
• Download Packages: ZIP files for multiple videos

Polling and Updates

• Frontend Polling: Every 2 seconds for real-time updates
• Backend Polling: Every 30 seconds to Google API
• Status Persistence: In-memory storage (consider Redis for scaling)
• Error Recovery: Exponential backoff with max 3 retries

Security and Cleanup

File Security

• Temporary Files: Automatic cleanup after download
• GCS Access: Service account with minimal permissions
• Local Storage: Isolated job folders with UUID naming
• Download Security: One-time download links with auto-deletion

Data Retention

• Job Data: Deleted after successful download
• Error Logs: Retained for debugging purposes
• User Data: Email tracking for analytics only
• File Cleanup: Complete removal including GCS resources

Scaling Considerations

Current Limitations

• In-Memory Storage: Job status stored in application memory
• Single Instance: No distributed processing support
• Concurrent Limits: Fixed 2-job processing limit

Scaling Solutions

• Redis Integration: Shared job status across instances
• Load Balancing: Multiple backend instances
• Database Storage: Persistent job history and analytics
• Queue Services: External queue management (RabbitMQ, SQS)

Monitoring and Observability

Metrics to Track

• Queue Length: Number of jobs waiting
• Processing Time: Average job completion time
• Success Rate: Percentage of successful generations
• Error Patterns: Common failure modes
• Resource Usage: CPU, memory, storage consumption

Logging Points

• Job Submission: User, prompt, parameters
• Status Changes: All state transitions with timestamps
• Error Events: Failures with stack traces
• Performance: Processing times and resource usage
• Cleanup Operations: File deletion and GCS operations

This lifecycle documentation provides a complete overview of how video generation jobs flow through the system, from submission to completion and cleanup.