dow-prod-tracker/docs/plans/gcs-storage-migration.md

GCS Storage Migration Plan

Problem

All uploaded images, videos, HLS segments, and thumbnails are stored on local disk at /data/uploads/. Video files (up to 500MB each) and HLS transcoded segments will fill the disk quickly.

Solution

Move file storage to a Google Cloud Storage (GCS) bucket with signed URLs for secure, direct-from-browser access. Processing (sharp, ffmpeg) stays local — only storage moves to GCS.

---

Architecture

Current Flow

Upload → sharp/ffmpeg process → write to /data/uploads → serve via /api/uploads/[...path] proxy

GCS Flow

Upload → sharp/ffmpeg process → upload to GCS bucket → store GCS object key in DB
Frontend → request signed URL from API → load directly from GCS (no server proxy)

---

What Changes

Component	Current	GCS
Image storage	/data/uploads/revisions/{id}/	gs://hp-tracker-media/revisions/{id}/
Video storage	Same local disk	Same GCS bucket
HLS segments	Local subdirectory	GCS bucket (uploaded after transcode)
File serving	/api/uploads/[...path] streams from disk	Signed URLs → browser loads direct from GCS
Thumbnails	Local disk	GCS bucket
Temp processing	N/A	Local /tmp for ffmpeg, deleted after upload to GCS

---

Files to Modify

Backend (4 files)

1. src/lib/services/upload-service.ts (main changes)

   • Replace writeFile() calls with GCS bucket.upload() or file.save()
   • Replace local URL generation (/api/uploads/revisions/...) with GCS object keys
   • Add getSignedUrl() helper for generating time-limited read URLs
   • Delete functions: replace unlink/rm with bucket.file().delete()
   • Lines affected: 86-149 (images), 200-290 (videos), 154-168 (image delete), 325-341 (video delete)

2. src/lib/services/video-service.ts

   • After transcodeToHLS() completes, upload all .ts segments + index.m3u8 to GCS
   • Rewrite playlist URLs in .m3u8 to reference GCS object paths (not local paths)
   • Delete local temp files after successful GCS upload
   • Lines affected: 137-189 (HLS transcode), 101-128 (thumbnail extraction)

3. src/lib/services/annotation-service.ts

   • Upload frame thumbnails to GCS instead of local disk
   • Lines affected: 105-149

4. src/app/api/uploads/[...path]/route.ts

   • Replace file streaming with signed URL redirect, OR
   • Convert to a signed-URL generator: GET /api/uploads/sign?key=revisions/xxx/file.mp4 → returns { url: "https://storage.googleapis.com/..." }
   • This preserves the auth check (only authenticated users get signed URLs)

Frontend (2 files, minor)

5. src/components/review/video-player.tsx / src/hooks/use-video-player.ts
   • HLS.js needs to load .m3u8 from a signed URL
   • If playlist contains relative segment URLs, those resolve against the playlist URL (GCS handles this)
   • May need xhrSetup config in HLS.js to handle CORS

Database (no schema change)

• The attachments JSON field already stores URLs as strings
• Just change from /api/uploads/revisions/... to GCS object keys like revisions/{id}/{file}
• Signed URLs are generated on-the-fly when the frontend requests them

---

HLS Playlist Strategy

The trickiest part. Two options:

Option A: Rewrite playlist at upload time (simpler)

• After transcoding, rewrite index.m3u8 so segment references use full GCS paths
• Upload the rewritten playlist to GCS
• Frontend gets one signed URL for the playlist, segments resolve as relative GCS URLs
• Downside: Segment URLs in the playlist aren't signed (bucket must allow public read for .ts files, or use uniform bucket-level access)

Option B: Signing proxy endpoint (more secure)

• Keep index.m3u8 with relative segment names
• Create /api/uploads/hls-playlist?key=revisions/{id}/video_hls/index.m3u8 endpoint
• This reads the playlist from GCS, replaces each segment filename with a signed URL, returns the modified playlist
• HLS.js loads this endpoint, gets signed URLs for every segment
• Downside: Every playlist fetch hits the server (but playlists are tiny)

Recommendation: Option A with a private bucket + signed URLs for the playlist. GCS signed URLs for the playlist will also cover the segments since HLS.js resolves segments relative to the playlist URL origin.

---

GCP Prerequisites

1. GCS Bucket: hp-tracker-media (or similar)

   • Location: same region as your server for low latency
   • Storage class: Standard
   • Access control: Uniform (not fine-grained)
   • Public access: Prevented (all access via signed URLs)

2. CORS Configuration on the bucket:

   [
     {
       "origin": ["https://hptracker.yourdomain.com"],
       "method": ["GET", "HEAD"],
       "responseHeader": ["Content-Type", "Content-Range", "Accept-Ranges"],
       "maxAgeSeconds": 3600
     }
   ]
   

3. Service Account:

   • Role: Storage Object Admin on the bucket
   • Download JSON key file
   • Mount as Docker secret or env var GOOGLE_APPLICATION_CREDENTIALS

4. Environment Variables (add to .env and docker-compose.yml):

   GCS_BUCKET=hp-tracker-media
   GCS_PROJECT_ID=your-gcp-project-id
   GOOGLE_APPLICATION_CREDENTIALS=/app/secrets/gcs-key.json
   

---

New Dependencies

npm install @google-cloud/storage

---

Migration Steps (for existing data)

1. Deploy the new code with GCS support
2. Run a one-time migration script to upload existing /data/uploads/ files to GCS
3. Update all revision.attachments JSON in the database to use GCS object keys
4. Verify everything works
5. Remove old local files and shrink the Docker volume

---

Estimated Effort

Phase	Work	Days
Image uploads + thumbnails	Replace writeFile with GCS upload, signed URL helper	1-2
Video uploads + HLS	Upload segments to GCS, playlist handling	1-2
Serving endpoint	Convert to signed URL generator	0.5
Annotation frames	Small change, same pattern as images	0.5
Deployment + bucket setup	GCS bucket, CORS, service account, Docker secrets	0.5
Data migration script	Upload existing files, update DB	0.5
Total		4-6 days

---

Rollback Plan

Keep the current local storage code behind a feature flag:

const USE_GCS = !!process.env.GCS_BUCKET;

If GCS is not configured, fall back to local disk (current behavior). This allows gradual rollout and easy rollback.