GCS Resumable Upload Pattern

For large file uploads that bypass the load balancer: browser initiates upload via backend, backend creates a GCS Resumable Session URI, returns it to the browser, and the browser uploads chunks directly to GCS — completely bypassing Apache and the load balancer.

Key Points

Browser uploads directly to GCS, not through Apache/LB — bypasses the LimitRequestBody constraint and LB timeouts
Backend creates the GCS session URI via service account credentials and returns it to browser
Session URI valid for 7 days (time to resume, not maximum upload time)
Chunk size must be a multiple of 256 KB; 8 MB is a practical default
308 Resume Incomplete = chunk accepted, send next chunk; 200/201 = upload complete
Network interruption recovery: PUT session_uri with Content-Range: bytes */{total_size} and 0-byte body → server returns current offset in Range header
LimitRequestBody 2147483648 (2 GB) needed in Apache only for the small /upload/init init request path

Details

Flow Overview

Browser → POST /api/v1/jobs/upload/init
              ↓
         Backend creates GCS Resumable Session URI
              ↓
         Returns { session_uri, upload_id } to browser
              ↓
Browser → PUT <session_uri> (chunk 1, bytes 0–8388607)
              ← 308 Resume Incomplete (continue)
Browser → PUT <session_uri> (chunk 2, bytes 8388608–16777215)
              ← 308 Resume Incomplete (continue)
...
Browser → PUT <session_uri> (final chunk)
              ← 200 OK or 201 Created (upload complete)
Browser → POST /api/v1/jobs/start { upload_id }

Backend: Create Session URI

from google.cloud import storage

def create_resumable_session(filename: str, content_type: str) -> str:
    client = storage.Client()
    bucket = client.bucket("my-bucket")
    blob = bucket.blob(f"uploads/{filename}")
    
    # Returns a resumable upload session URI
    session_uri = blob.create_resumable_upload_session(
        content_type=content_type,
        timeout=300,
    )
    return session_uri

Frontend: Chunk Upload

const CHUNK_SIZE = 8 * 1024 * 1024; // 8 MB — must be multiple of 256 KB

async function uploadChunked(sessionUri: string, file: File): Promise<void> {
  let offset = 0;
  
  while (offset < file.size) {
    const chunk = file.slice(offset, offset + CHUNK_SIZE);
    const isLast = offset + CHUNK_SIZE >= file.size;
    
    const response = await fetch(sessionUri, {
      method: "PUT",
      headers: {
        "Content-Range": `bytes ${offset}-${offset + chunk.size - 1}/${file.size}`,
        "Content-Type": file.type,
      },
      body: chunk,
    });
    
    if (response.status === 308) {
      // Chunk accepted — continue
      offset += chunk.size;
    } else if (response.status === 200 || response.status === 201) {
      // Upload complete
      break;
    } else {
      throw new Error(`Upload failed: ${response.status}`);
    }
  }
}

Resuming After Network Interruption

async function getUploadOffset(sessionUri: string, totalSize: number): Promise<number> {
  const response = await fetch(sessionUri, {
    method: "PUT",
    headers: {
      "Content-Range": `bytes */${totalSize}`,
      "Content-Length": "0",
    },
    body: null,
  });
  
  if (response.status === 308) {
    const range = response.headers.get("Range"); // "bytes=0-8388607"
    return parseInt(range!.split("-")[1]) + 1;
  }
  return 0; // No chunks received yet
}

Apache Configuration

# Only needed for the /upload/init endpoint (small JSON payload)
# The actual file chunks go directly to GCS, bypassing Apache
<Location /video-accessibility/api/v1/jobs/upload>
    LimitRequestBody 2147483648
</Location>

The 2147483648 value is 2 GB in bytes — for the small JSON init request this is academic, but Apache's default limit (usually 1 MB) would block even the metadata request on some configs.

wiki/architecture/optical-dev-server-deploy — Apache configuration on optical-dev, LimitRequestBody context
wiki/architecture/gcp-deployment-lb-timeout — why GCP LB timeouts make direct-to-GCS uploads necessary for large files

Sources

daily/2026-04-30.md — Session 11:28, GCS resumable upload implementation for video-accessibility

4.5 KiB Raw Blame History Unescape Escape