Celery Queue–Worker Specialization

When Celery tasks are routed to a named queue (e.g., tts), ONLY the container consuming that queue will process them. The main/orchestrator worker will not pick up tasks from specialised queues. Changing code in a specialised worker requires rebuilding THAT container, not the main worker.

Key Points

Each Celery worker listens to one or more named queues — it will NOT process tasks from other queues
The orchestrator (accessible-video-worker) only processes tasks on its queue — it will never pick up tasks from tts queue
When tasks queue but completion counter stays at 0, always check whether the correct specialised worker is running and has the latest code
Fixing a bug in the TTS worker requires rebuilding tts-worker, NOT accessible-video-worker
This is intentional design: specialised workers optimise for specific hardware/dependency requirements

Details

Queue Assignment in Video Accessibility

Queue: celery (default)     → accessible-video-worker (orchestrator)
Queue: tts                  → tts-worker (TTS synthesis)
Queue: transcription        → transcription-worker (Whisper)

A task routed to tts queue:

@celery.task(queue="tts")
def synthesize_speech(text: str, voice: str) -> str:
    ...

This task ONLY runs in the container started with --queues tts:

celery -A app worker --queues tts --concurrency 2

Debugging Stuck Tasks

When tasks enqueue but completions stay at 0:

# 1. Check which containers are running
docker compose ps

# 2. Check the correct worker is consuming its queue
docker compose logs tts-worker --tail 50

# 3. Check Redis queue depth
docker compose exec redis redis-cli LLEN tts

# 4. Verify the worker is registered for the task
docker compose exec tts-worker celery -A app inspect registered

Rebuild the Right Container

# Bug is in TTS synthesis code:
docker compose build tts-worker
docker compose up -d tts-worker

# NOT:
docker compose build accessible-video-worker  # wrong — won't fix tts tasks

When Tasks Queue But Don't Complete

Symptom	Likely cause
Tasks in Redis `tts` queue, zero completions	`tts-worker` not running or on old code
Tasks in Redis `celery` queue, zero completions	`accessible-video-worker` not running
Tasks complete but error every time	Deterministic error — see wiki/concepts/celery-redis-queue-flush-on-deterministic-error

wiki/concepts/celery-redis-queue-flush-on-deterministic-error — flushing Redis when the right worker is running but tasks keep failing
wiki/tech-patterns/redis-celery-worker-queue — Redis + Celery full pattern reference for Oliver projects

Sources

daily/2026-04-30.md — Session 21:14, TTS tasks queuing but completions at 0 — wrong worker rebuilt

3.1 KiB Raw Permalink Blame History Unescape Escape