video-accessibility/DEPLOYMENT.md

# Accessible Video Platform - Production Deployment Guide

> **Canonical source moved:** See [docs/project/runbook.md](docs/project/runbook.md) for the current deployment runbook and [docs/project/infrastructure.md](docs/project/infrastructure.md) for the infrastructure inventory. This file is kept for historical reference.


## =<3D> Table of Contents
- [Overview](#overview)
- [Server Requirements](#server-requirements)
- [Pre-Deployment Checklist](#pre-deployment-checklist)
- [Initial Setup](#initial-setup)
- [Deployment Steps](#deployment-steps)
- [Post-Deployment Verification](#post-deployment-verification)
- [Maintenance & Operations](#maintenance--operations)
- [Troubleshooting](#troubleshooting)

---

## <<3C> Overview

This guide covers deploying the Accessible Video Processing Platform to a production server at `ai-sandbox.oliver.solutions`.

**Architecture:**
- **Frontend**: React SPA served by Apache at `/video-accessibility`
- **Backend API**: Docker container proxied by Apache at `/video-accessibility-back`
- **Worker**: Docker container for Celery background processing
- **MongoDB**: Docker container for database
- **Redis**: Docker container for queue/cache

**Server Details:**
- **Host**: ai-sandbox.oliver.solutions (GCP VM)
- **Resources**: 32GB RAM, 8 CPU cores
- **OS**: Linux with Apache 2.4+
- **SSL**: Wildcard certificate configured

---

## =<3D> Server Requirements

### Software Requirements
- Docker 20.10+
- Docker Compose 1.29+
- Apache 2.4+ with modules:
  - `mod_rewrite`
  - `mod_proxy`
  - `mod_proxy_http`
  - `mod_proxy_wstunnel`
  - `mod_headers`
- Node.js 20+ and npm (for building frontend)
- Git (for code deployment)

### Apache Modules Setup
```bash
sudo a2enmod rewrite proxy proxy_http proxy_wstunnel headers
sudo systemctl restart apache2
```

### Directory Structure
```
/opt/accessible-video/
 docker-compose.yml
 docker-compose.prod.yml
 .env.production
 backend/               # Git clone
 frontend/              # Git clone
 secrets/
    gcp-credentials.json
 scripts/
    deploy.sh
    build-frontend.sh
    mongodb-init.js
 volumes/              # Created by Docker
     mongodb-data/
     redis-data/
     logs/
```

---

##  Pre-Deployment Checklist

### 1. Secrets & Credentials
- [ ] Google Cloud credentials JSON file ready
- [ ] Generate secure JWT secret (64 characters)
  ```bash
  openssl rand -hex 32
  ```
- [ ] Gemini API key obtained
- [ ] (Optional) SendGrid API key for emails
- [ ] (Optional) ElevenLabs API key for TTS

### 2. DNS & SSL
- [ ] DNS for `ai-sandbox.oliver.solutions` configured
- [ ] Wildcard SSL certificate installed in Apache
- [ ] Certificate includes `*.ai-sandbox.oliver.solutions`

### 3. GCP Setup
- [ ] GCS bucket `accessible-video` created
- [ ] Service account has Storage Admin role
- [ ] Service account JSON key downloaded

### 4. Firewall & Ports
- [ ] Port 443 (HTTPS) open
- [ ] Port 80 (HTTP) open (for redirect to HTTPS)
- [ ] Port 8000 accessible from localhost only (Docker API)

---

## =<3D> Initial Setup

### Step 1: Clone Repositories

```bash
# Create project directory
sudo mkdir -p /opt/video-accessibility
cd /opt/video-accessibility

# Clone backend
git clone <your-backend-repo-url> backend

# Clone frontend
git clone <your-frontend-repo-url> frontend

# Set ownership
sudo chown -R $USER:$USER /opt/video-accessibility
```

### Step 2: Create Secrets Directory

```bash
# Create secrets directory
mkdir -p /opt/video-accessibility/secrets

# Copy GCP credentials
cp /path/to/gcp-credentials.json /opt/video-accessibility/secrets/gcp-credentials.json

# Set restrictive permissions
chmod 600 /opt/video-accessibility/secrets/gcp-credentials.json
```

### Step 3: Configure Environment Variables

```bash
# Copy template
cp .env.production.template .env.production

# Edit with secure values
nano .env.production
```

**Required changes in `.env.production`:**
```bash
# Generate JWT secret (64 characters)
JWT_SECRET=<64-char-random-string>

# Update with your values
GCP_PROJECT_ID=optical-414516
GCS_BUCKET=accessible-video
GEMINI_API_KEY=<your-gemini-key>

# Optional
SENDGRID_API_KEY=<your-sendgrid-key>
ELEVENLABS_API_KEY=<your-elevenlabs-key>
```

```bash
# Set secure permissions
chmod 600 .env.production
```

### Step 4: Configure Frontend

```bash
# Frontend environment is already configured in frontend/.env.production
# Verify the API URL is correct:
cat frontend/.env.production
```

Should contain:
```
VITE_API_BASE_URL=https://ai-sandbox.oliver.solutions/video-accessibility-back
```

### Step 5: Configure Apache

```bash
# Add configuration to your VirtualHost
sudo nano /etc/apache2/sites-available/ai-sandbox.oliver.solutions-ssl.conf

# Copy configuration from apache-config-snippet.conf
# Add inside the <VirtualHost *:443> block

# Test configuration
sudo apache2ctl configtest

# Restart Apache
sudo systemctl restart apache2
```

---

## =<3D> Deployment Steps

### Option A: Automated Deployment (Recommended)

```bash
cd /opt/accessible-video

# Make scripts executable
chmod +x scripts/*.sh

# Run full deployment
./scripts/deploy.sh
```

This script will:
1. Pull latest code
2. Build Docker images
3. Start containers
4. Build and deploy frontend
5. Run database migrations
6. Verify deployment

### Option B: Manual Deployment

#### 1. Build and Start Backend Services

```bash
cd /opt/video-accessibility

# Load environment variables
export $(cat .env.production | grep -v '^#' | xargs)

# Build images
docker compose -f docker-compose.yml -f docker-compose.prod.yml build

# Start services
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Check status
docker compose ps
```

#### 2. Initialize Database

```bash
# Wait for MongoDB to be ready (about 10 seconds)
sleep 10

# Initialize database
docker compose exec mongodb mongosh < scripts/mongodb-init.js

# Run migrations
docker compose exec api python migrate.py

# Create initial admin user
docker compose exec api python create_test_users.py
```

#### 3. Build and Deploy Frontend

```bash
# Run frontend build script
./scripts/build-frontend.sh
```

Or manually:
```bash
cd frontend

# Install dependencies
npm ci --only=production

# Build
npm run build

# Deploy to Apache
sudo mkdir -p /var/www/html/video-accessibility
sudo rm -rf /var/www/html/video-accessibility/*
sudo cp -r dist/* /var/www/html/video-accessibility/
sudo chown -R www-data:www-data /var/www/html/video-accessibility
sudo chmod -R 755 /var/www/html/video-accessibility

cd ..
```

---

##  Post-Deployment Verification

### 1. Check Docker Containers

```bash
cd /opt/accessible-video
docker-compose ps
```

All containers should show as "healthy" or "Up".

### 2. Test Backend API

```bash
# Health check
curl https://ai-sandbox.oliver.solutions/video-accessibility-back/health

# Should return: {"status":"healthy","version":"1.0.0"}
```

### 3. Test Frontend

Visit in browser:
```
https://ai-sandbox.oliver.solutions/video-accessibility
```

You should see the login page.

### 4. Test WebSocket Connection

Check browser console for WebSocket connection (after logging in).

### 5. Create Test Job

1. Log in as admin user
2. Upload a test video
3. Verify job appears in the list
4. Check worker logs:
   ```bash
   docker-compose logs -f worker
   ```

---

## =' Maintenance & Operations

### Viewing Logs

```bash
# View all logs
docker compose logs -f

# View specific service
docker compose logs -f api
docker compose logs -f worker

# View last 100 lines
docker compose logs --tail=100 api
```

### Restarting Services

```bash
# Restart all
docker compose restart

# Restart specific service
docker compose restart api
docker compose restart worker
```

### Updating Application

```bash
# Pull latest code (as your user, not sudo)
cd /opt/video-accessibility
git pull origin main
cd backend && git pull origin main && cd ..
cd frontend && git pull origin main && cd ..

# Run full deployment (requires sudo)
sudo ./scripts/full-deploy.sh
```

### Updating Frontend Only

```bash
cd /opt/video-accessibility
./scripts/build-frontend.sh
```

### Database Backup

```bash
# Backup MongoDB
docker compose exec mongodb mongodump --out=/data/backup

# Copy backup from container
docker cp accessible-video-mongodb:/data/backup ./mongodb-backup-$(date +%Y%m%d)
```

### Database Restore

```bash
# Copy backup to container
docker cp ./mongodb-backup accessible-video-mongodb:/data/restore

# Restore
docker compose exec mongodb mongorestore /data/restore
```

### Monitoring Resource Usage

```bash
# Check Docker container resource usage
docker stats

# Check disk usage
docker system df

# Clean up unused images/containers
docker system prune -a
```

---

## = Troubleshooting

### Issue: Containers Won't Start

**Check logs:**
```bash
docker compose logs
```

**Common causes:**
- MongoDB authentication failure <20> Check `MONGODB_ROOT_PASSWORD`
- Redis connection failure <20> Check if Redis container is running
- Port 8000 already in use <20> Check for other processes on port 8000

### Issue: 502 Bad Gateway on Backend

**Diagnosis:**
```bash
# Check if API container is running
docker-compose ps api

# Check API logs
docker-compose logs api

# Test API directly
curl http://localhost:8000/health
```

**Solutions:**
- API container not started <20> `docker-compose up -d api`
- API crashed <20> Check logs for errors
- Wrong port in Apache config <20> Should proxy to `localhost:8000`

### Issue: Frontend Shows 404

**Check:**
```bash
# Verify files exist
ls -la /var/www/html/video-accessibility/

# Check Apache logs
sudo tail -f /var/log/apache2/ai-sandbox-error.log
```

**Solutions:**
- Files not deployed <20> Run `./scripts/build-frontend.sh`
- Apache alias incorrect <20> Check `apache-config-snippet.conf`
- Permissions wrong <20> `sudo chown -R www-data:www-data /var/www/html/video-accessibility`

### Issue: WebSocket Connection Fails

**Check:**
```bash
# Verify mod_proxy_wstunnel is enabled
apache2ctl -M | grep proxy_wstunnel
```

**Solution:**
```bash
sudo a2enmod proxy_wstunnel
sudo systemctl restart apache2
```

### Issue: Worker Not Processing Jobs

**Check:**
```bash
# View worker logs
docker compose logs -f worker

# Check Redis connection
docker compose exec worker python -c "import redis; r=redis.Redis(host='redis'); print(r.ping())"

# Check Celery status
docker compose exec worker celery -A celery_worker inspect active
```

**Solutions:**
- Worker not listening to correct queues <20> Check Dockerfile CMD
- Redis connection issue <20> Check `REDIS_URL` environment variable
- GCP credentials missing <20> Check `/secrets/gcp-credentials.json` is mounted

### Issue: Upload Fails / GCS Error

**Check:**
```bash
# Test GCS credentials
docker compose exec api python -c "
from google.cloud import storage
client = storage.Client()
print(list(client.list_buckets()))
"
```

**Solutions:**
- Credentials file not found <20> Check `secrets/gcp-credentials.json` exists
- Wrong project ID <20> Check `GCP_PROJECT_ID` in `.env.production`
- Permissions issue <20> Service account needs Storage Admin role

### Issue: Database Connection Errors

**Check:**
```bash
# Test MongoDB connection
docker-compose exec api python -c "
from pymongo import MongoClient
import os
client = MongoClient(os.environ['MONGODB_URI'])
print(client.server_info())
"
```

**Solutions:**
- Wrong credentials <20> Check `MONGODB_ROOT_PASSWORD`
- MongoDB not started <20> `docker-compose up -d mongodb`
- Connection string incorrect <20> Should be `mongodb://admin:password@mongodb:27017/...`

---

## =<3D> Support & Additional Resources

- **Project Repository**: `<your-repo-url>`
- **Issues**: Create an issue in the repository
- **Docker Documentation**: https://docs.docker.com/
- **Apache Documentation**: https://httpd.apache.org/docs/2.4/

---

## =<3D> Quick Reference Commands

```bash
# Full deployment
cd /opt/video-accessibility && ./scripts/deploy.sh

# View logs
docker compose logs -f [service]

# Restart service
docker compose restart [service]

# Update frontend only
./scripts/build-frontend.sh

# Check container status
docker compose ps

# View resource usage
docker stats

# Backup database
docker compose exec mongodb mongodump --out=/data/backup

# Access MongoDB shell
docker compose exec mongodb mongosh

# Run migrations
docker compose exec api python migrate.py
```

---

**Last Updated**: 2025-01-10
**Version**: 1.0.0