---
title: "Celery Worker Restart Orphans In-Flight Tasks"
description: "Tasks in ai_processing state during a worker restart are permanently orphaned — no auto-recovery; requires manual reprocess-job/{id} after each deploy"
tags: [celery, deploy, video-accessibility, worker, mongodb]
source: daily/2026-05-08.md
created: 2026-05-08
updated: 2026-05-08
---

# Celery Worker Restart Orphans In-Flight Tasks

## The Problem

When a Celery worker container is restarted (e.g., during a deploy), any tasks currently executing are killed mid-flight. The task function body never completes — but the MongoDB job document was already set to `ai_processing` (or a similar in-progress state) before Celery picked up the task. After restart, the worker starts fresh with no memory of the killed task. The job is permanently stuck in its in-progress state with no error logged.

```
Job A → state: ai_processing (set before task enqueue)
       → Celery picks up task
       → Worker container restarts (deploy)
       → Task killed
       → Job A stays in state: ai_processing — FOREVER
```

There is **no automatic recovery**. Celery does not re-enqueue killed tasks by default. MongoDB has no TTL on the in-progress state.

## Detection

Before any deploy:
```bash
# Count jobs stuck in-progress
db.jobs.countDocuments({ status: "ai_processing" })
# or check the admin panel for stuck jobs
```

## Recovery

After deploy, manually re-trigger stuck jobs:
```bash
curl -X POST https://<host>/api/admin/reprocess-job/<job_id>
```

Or for bulk recovery:
```python
# Admin script
stuck = db.jobs.find({"status": "ai_processing"})
for job in stuck:
    requests.post(f"/api/admin/reprocess-job/{job['_id']}")
```

## Deploy Checklist

1. Check `db.jobs.countDocuments({ status: "ai_processing" })` before deploy
2. Wait for in-flight count to reach 0 OR notify job owners
3. Deploy
4. After deploy: re-trigger any jobs that were stuck

## Mitigation Options

- **Graceful shutdown**: `docker compose stop` (not `kill`) gives workers time to finish current tasks before shutting down (up to `--time` seconds)
- **Task timeout + retry**: set `soft_time_limit` and `autoretry_for` on tasks to auto-retry if killed
- **State guard**: set job to `ai_processing` only AFTER the task starts executing (not before enqueue) — prevents permanent stuck state if enqueue never reaches the worker

## Related

- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — related: flushing Redis queue for deterministic errors
- [[wiki/tech-patterns/redis-celery-worker-queue]] — full pipeline architecture
