b3ace22009
Heavy pipeline tasks (ingest, translate, render, rerender) now dispatch to a Cloud Run Job (va-worker) instead of local Celery workers. optical-dev runs only api + lightweight worker (notify/embed) within its 2-CPU budget. - backend/app/tasks/runner.py — Cloud Run Job entrypoint - backend/app/services/cloud_run_dispatch.py — replaces .delay() for heavy tasks - backend/Dockerfile.cloudrun — Cloud Run worker image (ffmpeg included) - docker-compose.optical-dev.yml — 2-CPU safe overrides, disables heavy workers - cloudbuild.yaml — builds va-worker image and updates Cloud Run Job - deploy-dev.sh — uses 3-file compose, builds only api+worker locally - routes_jobs, routes_admin_production, ingest_and_ai, translate_and_synthesize — all dispatch sites updated to use cloud_run_dispatch.dispatch() USE_CELERY_FALLBACK=true in .env.local to use Celery locally during dev. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
55 lines
2.1 KiB
Text
55 lines
2.1 KiB
Text
# =============================================================================
|
|
# Cloud Run Job image — va-worker
|
|
#
|
|
# Reuses the multi-stage base from Dockerfile.
|
|
# Entrypoint: python -m app.tasks.runner --task <name> --job-id <id>
|
|
#
|
|
# Build:
|
|
# docker build -f backend/Dockerfile.cloudrun -t va-worker backend/
|
|
# =============================================================================
|
|
|
|
# ── Stage 1: Builder ─────────────────────────────────────────────────────────
|
|
FROM python:3.11-slim AS builder
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
build-essential curl \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN pip install --no-cache-dir poetry==1.8.3
|
|
|
|
WORKDIR /app
|
|
COPY pyproject.toml poetry.lock ./
|
|
RUN poetry config virtualenvs.create false \
|
|
&& poetry install --no-interaction --no-ansi --only main
|
|
|
|
# ── Stage 2: Runtime ─────────────────────────────────────────────────────────
|
|
FROM python:3.11-slim AS runtime
|
|
|
|
# ffmpeg required for video rendering tasks
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
ffmpeg \
|
|
tini \
|
|
curl \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Copy installed packages from builder
|
|
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
|
|
COPY --from=builder /usr/local/bin /usr/local/bin
|
|
|
|
WORKDIR /app
|
|
COPY . .
|
|
|
|
# Non-root user for security
|
|
RUN groupadd -r worker && useradd -r -g worker worker \
|
|
&& chown -R worker:worker /app
|
|
USER worker
|
|
|
|
# Cloud Run Jobs: no persistent HTTP port needed.
|
|
# Cloud Run passes CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_COUNT env vars.
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
PYTHONPATH=/app
|
|
|
|
ENTRYPOINT ["tini", "--", "python", "-m", "app.tasks.runner"]
|
|
# Args are injected per-execution via Cloud Run Job overrides:
|
|
# --task ingest|translate|render|rerender --job-id <id> [--language <lang>] ...
|