---
title: "Connection: Celery Prefork × faster_whisper — Memory Stacking"
connects:
  - "concepts/celery-prefork-pool-startup-memory"
  - "concepts/faster-whisper-startup-memory"
sources:
  - "daily/2026-04-30.md"
created: 2026-04-30
updated: 2026-04-30
---

# Connection: Celery Prefork × faster_whisper — Memory Stacking

## The Connection

Two independent startup-memory behaviours combine multiplicatively when `faster_whisper` is imported inside a Celery worker module:

1. **Celery prefork** forks ALL `CONCURRENCY` worker processes at `celery worker` start — each is a full Python interpreter with all imports loaded.
2. **faster_whisper** loads the full transcription model into RAM at import time (when `WhisperModel(...)` is called at module level or in a module-level `@worker_init` signal handler).

Result: `CONCURRENCY=4` with a 400 MB Whisper model = **1.6 GB** consumed before the first transcription task is dequeued.

## Key Insight

> Neither behaviour is a bug in isolation — the danger is invisible until they are combined in the same container.

The `faster-whisper-startup-memory` article documents the per-container model loading cost. The `celery-prefork-pool-startup-memory` article documents the per-worker process forking cost. When they stack, the formula becomes:

```
total_startup_RAM = CONCURRENCY × (base_worker_MB + model_size_MB)
```

Example with `large-v3` model (~1.5 GB) and `CONCURRENCY=4`:

```
4 × (80 MB interpreter + 1500 MB model) = 6.3 GB before first task
```

A container with a 4 GB memory limit is OOM-killed before it processes anything.

## Evidence

- Session 21:37 (2026-04-30): ffmpeg-worker with `CONCURRENCY=20`, ~120 MB/process → 2.4 GB, container OOM-killed, 15-minute pipeline stall
- The stall was compounded because Celery silently retries tasks that were in-flight when the worker died, creating a second wave of OOM on restart

## Solutions

### Option A — Reduce concurrency to match model size

```
CONCURRENCY = floor(container_memory_MB / (base_MB + model_MB))
```

### Option B — Separate transcription into its own single-worker container

Keep `CONCURRENCY=1` for the whisper worker, scale by adding containers, not by increasing CONCURRENCY. Each container has exactly one model copy.

### Option C — Load model lazily (inside the task, not at import)

```python
_model = None

@app.task
def transcribe(audio_path: str):
    global _model
    if _model is None:
        _model = WhisperModel("large-v3")
    return _model.transcribe(audio_path)
```

Downside: first task in each process pays the load latency (~5–15 s). Subsequent tasks in the same process reuse the loaded model.

### Option D — Use `solo` or `threads` pool

`CELERY_POOL=solo` runs tasks in the main process with no forking — only one model copy regardless of logical concurrency. Appropriate for GPU workers where parallelism is handled at the GPU level.

## Related Concepts

- [[wiki/concepts/celery-prefork-pool-startup-memory]] — Celery fork-all-at-startup behaviour
- [[wiki/concepts/faster-whisper-startup-memory]] — model loaded at container start
- [[wiki/concepts/celery-queue-worker-specialization]] — isolating whisper work to dedicated containers
- [[wiki/concepts/docker-compose-cpu-limits-env]] — setting memory limits in Compose

## Sources

- [[daily/2026-04-30.md]] — Session 21:37, Celery ffmpeg-worker OOM; identified as combined prefork + model-loading issue
