---
title: "FastAPI + Docker — asyncio Task Loss on Restart: UI Deadlock and Startup Auto-Recovery"
aliases: [asyncio-container-restart, stale-job-recovery, fastapi-startup-recovery]
tags: [fastapi, asyncio, docker, barclays, gotcha]
created: 2026-05-15
updated: 2026-05-15
---

# FastAPI + Docker — asyncio Task Loss on Restart: UI Deadlock and Startup Auto-Recovery

## The Problem

`asyncio.create_task()` tasks are **killed silently** when a Docker container restarts. The task disappears from the event loop but any DB records it was writing remain in their previous state (e.g. `status = "processing"`).

This creates a cascading deadlock:

1. Container restarts mid-job → task dies, DB row stays `status = "processing"`
2. UI hides the "Process" button when any job is in-progress (standard guard)
3. User cannot trigger a new run to recover — they are locked out

No error is logged. The UI shows a spinner forever.

---

## Smart-Resume Pattern (Startup Auto-Recovery)

Trigger a recovery coroutine from the FastAPI `lifespan` startup hook. The recovery:
1. Finds all jobs still in `processing` state
2. Marks them `failed` (or re-queues them for retry)
3. Optionally skips already-completed sub-tasks (smart-resume) to avoid re-paying API costs

```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
import asyncio

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Run at startup — recover stale jobs before accepting traffic
    asyncio.create_task(recover_stale_jobs())
    yield
    # cleanup on shutdown (optional)

async def recover_stale_jobs():
    """Mark in-progress jobs that survived a restart as failed, or re-queue them."""
    async with get_db_session() as db:
        stale = await db.execute(
            select(Job).where(Job.status == "processing")
        )
        for job in stale.scalars():
            # Smart-resume: skip docs already parsed to save API cost
            already_done = await db.execute(
                select(Document).where(
                    Document.job_id == job.id,
                    Document.status == "completed"
                )
            )
            pending = [d for d in job.documents if d not in already_done]
            if not pending:
                job.status = "completed"
            else:
                job.status = "failed"  # or re-queue for retry
        await db.commit()

app = FastAPI(lifespan=lifespan)
```

---

## Smart-Resume vs Full Retry

| Strategy | When to use | Cost |
|----------|-------------|------|
| Mark as `failed` + let user retry | Simple; user retriggers manually | User pays full re-run cost |
| Smart-resume: skip completed sub-tasks | Many documents, expensive API (e.g. LlamaExtract) | Cheaper; re-runs only pending docs |
| Auto-requeue | High-SLA background jobs | Complex; risk of double-processing |

Smart-resume requires each document/chunk to have its own `status` column so recovery can skip `completed` ones.

> [!warning] `parsing` ≠ `parsed` — do not skip interrupted documents
> Documents in `parsing` state (interrupted mid-parse by the restart) must be **reset to `pending`**, not skipped. Only `parsed`/`completed` documents (finished before the restart) are safe to skip. A `parsing`-interrupted document may have partial or missing output — re-running it is mandatory.

---

## UI Deadlock Root Cause

The guard `if any job is active → hide Process button` is correct in steady state. The bug is that it has no escape hatch for the orphaned-task scenario.

**Fix options:**
- Add a manual "Force reset" admin action that marks all stale jobs as `failed`
- Add a timeout: jobs stuck in `processing` for > N minutes are auto-failed by a cron/background task
- Use the startup recovery above — recovery fires before the first user sees the UI

---

## Related

- [[wiki/concepts/celery-inflight-tasks-lost-on-restart|celery-inflight-tasks-lost-on-restart]] — same pattern for Celery workers
- [[wiki/concepts/fastapi-lifespan-locals-app-state|fastapi-lifespan-locals-app-state]] — lifespan scope gotcha
- [[wiki/client-knowledge/barclays|barclays]] — KB processing pipeline where this deadlock was observed
