89 lines
3.1 KiB
Markdown
89 lines
3.1 KiB
Markdown
---
|
||
title: "Celery Queue–Worker Specialization"
|
||
aliases: [celery-named-queues, celery-worker-routing, celery-task-queue-assignment]
|
||
tags: [celery, python, worker, docker, gotcha, deployment]
|
||
sources:
|
||
- "daily/2026-04-30.md"
|
||
created: 2026-04-30
|
||
updated: 2026-04-30
|
||
---
|
||
|
||
# Celery Queue–Worker Specialization
|
||
|
||
When Celery tasks are routed to a named queue (e.g., `tts`), ONLY the container consuming that queue will process them. The main/orchestrator worker will not pick up tasks from specialised queues. Changing code in a specialised worker requires rebuilding THAT container, not the main worker.
|
||
|
||
## Key Points
|
||
|
||
- Each Celery worker listens to one or more named queues — it will NOT process tasks from other queues
|
||
- The orchestrator (`accessible-video-worker`) only processes tasks on its queue — it will never pick up tasks from `tts` queue
|
||
- When tasks queue but completion counter stays at 0, always check whether the correct specialised worker is running and has the latest code
|
||
- Fixing a bug in the TTS worker requires rebuilding `tts-worker`, NOT `accessible-video-worker`
|
||
- This is intentional design: specialised workers optimise for specific hardware/dependency requirements
|
||
|
||
## Details
|
||
|
||
### Queue Assignment in Video Accessibility
|
||
|
||
```
|
||
Queue: celery (default) → accessible-video-worker (orchestrator)
|
||
Queue: tts → tts-worker (TTS synthesis)
|
||
Queue: transcription → transcription-worker (Whisper)
|
||
```
|
||
|
||
A task routed to `tts` queue:
|
||
```python
|
||
@celery.task(queue="tts")
|
||
def synthesize_speech(text: str, voice: str) -> str:
|
||
...
|
||
```
|
||
|
||
This task ONLY runs in the container started with `--queues tts`:
|
||
```bash
|
||
celery -A app worker --queues tts --concurrency 2
|
||
```
|
||
|
||
### Debugging Stuck Tasks
|
||
|
||
When tasks enqueue but completions stay at 0:
|
||
|
||
```bash
|
||
# 1. Check which containers are running
|
||
docker compose ps
|
||
|
||
# 2. Check the correct worker is consuming its queue
|
||
docker compose logs tts-worker --tail 50
|
||
|
||
# 3. Check Redis queue depth
|
||
docker compose exec redis redis-cli LLEN tts
|
||
|
||
# 4. Verify the worker is registered for the task
|
||
docker compose exec tts-worker celery -A app inspect registered
|
||
```
|
||
|
||
### Rebuild the Right Container
|
||
|
||
```bash
|
||
# Bug is in TTS synthesis code:
|
||
docker compose build tts-worker
|
||
docker compose up -d tts-worker
|
||
|
||
# NOT:
|
||
docker compose build accessible-video-worker # wrong — won't fix tts tasks
|
||
```
|
||
|
||
### When Tasks Queue But Don't Complete
|
||
|
||
| Symptom | Likely cause |
|
||
|---------|-------------|
|
||
| Tasks in Redis `tts` queue, zero completions | `tts-worker` not running or on old code |
|
||
| Tasks in Redis `celery` queue, zero completions | `accessible-video-worker` not running |
|
||
| Tasks complete but error every time | Deterministic error — see [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] |
|
||
|
||
## Related Concepts
|
||
|
||
- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — flushing Redis when the right worker is running but tasks keep failing
|
||
- [[wiki/tech-patterns/redis-celery-worker-queue]] — Redis + Celery full pattern reference for Oliver projects
|
||
|
||
## Sources
|
||
|
||
- [[daily/2026-04-30.md]] — Session 21:14, TTS tasks queuing but completions at 0 — wrong worker rebuilt
|