---
title: "Celery In-Flight Tasks Lost on Worker Restart"
aliases: [celery-restart-task-loss, celery-deploy-lost-tasks, celery-inflight]
tags: [celery, python, deploy, video-accessibility, ops]
sources:
  - "daily/2026-05-08.md"
created: 2026-05-08
updated: 2026-05-08
---

# Celery In-Flight Tasks Lost on Worker Restart

Tasks in the `ai_processing` (or any custom) state at the time of a Celery worker restart are permanently lost. Celery does not automatically recover them — there is no built-in checkpoint or re-queue mechanism for in-flight tasks.

## What Happens

1. Worker receives a task and begins execution → sets job status to `ai_processing` in MongoDB
2. Deploy restarts the worker container mid-execution
3. The task disappears from Celery — no retry, no error callback
4. MongoDB document stays in `ai_processing` forever
5. User sees a job that is "stuck" — UI shows processing spinner indefinitely

## No Automatic Recovery

```python
# Celery ACKS_LATE / REJECT_ON_WORKER_LOST help with QUEUED tasks
# but NOT with tasks that are actively executing when the worker dies
app.conf.task_acks_late = True           # only protects pre-execution
app.conf.task_reject_on_worker_lost = True  # same — only pre-execution
```

These settings protect tasks that were **dequeued but not yet started**. Once execution has begun (status → `ai_processing`), the task is acknowledged and removing it from the queue — a kill at this point is permanent loss.

## Recovery Pattern

After any deploy that may have interrupted active workers:

```bash
# 1. Find stuck jobs
db.jobs.find({ status: "ai_processing" })

# 2. Manually re-trigger each one via the API
curl -X POST https://api.example.com/api/v1/admin/reprocess-job/{job_id}

# 3. Verify new task enqueued
celery -A app.worker inspect active
```

## Post-Deploy Checklist

Add to deployment runbook:

```
[ ] Check for jobs in ai_processing state before deploying:
    db.jobs.countDocuments({ status: "ai_processing" })
[ ] If count > 0: wait for jobs to complete OR accept re-processing cost
[ ] After deploy: check for stuck jobs (status = ai_processing, updated_at > 10 min ago)
[ ] Re-trigger any stuck jobs via /admin/reprocess-job/{id}
```

## Architectural Note

This is a known limitation of the current architecture. Proper solutions (none yet implemented):

- **Pre-drain**: before restart, stop accepting new tasks and wait for active tasks to complete (`celery -A app.worker control shutdown` after tasks drain)
- **Checkpoint/resume**: save task progress to MongoDB so it can be resumed after restart
- **Outbox pattern**: record "task dispatched" before sending to Celery; a reconciler re-enqueues tasks that were dispatched but not completed

## Related Concepts

- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — Redis queue flush for stuck tasks from deterministic errors
- [[wiki/concepts/celery-queue-worker-specialization]] — Named queue / worker specialization
- [[wiki/concepts/celery-prefork-pool-startup-memory]] — Memory behaviour of Celery prefork workers

## Sources

- [[daily/2026-05-08.md]] — Identified during video-accessibility pipeline video processing deploy; tasks in `ai_processing` state were lost when the worker container was restarted as part of the TTS migration deploy
