Vadym Samoilenko a3b300b76a docs: add canonical documentation + audit cleanup

- AGENTS.md: canonical project entry point (Quick Nav, pipeline, constraints)
- docs/: complete docs tree — architecture, API spec, DB schema, infra,
  runbook, requirements, tech stack, principles, reference ADRs, guides,
  tasks backlog, testing strategy
- tests/README.md: test commands, structure, known gaps
- README.md / CLAUDE.md / DEPLOYMENT.md: updated with canonical doc links
- .archive/: backup of pre-documentation-pipeline originals
- backend/uv.lock: uv dependency lockfile
- Delete committed __pycache__ .pyc files (should have been gitignored)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-29 14:22:51 +01:00

1.6 KiB

Raw Blame History

ADR-001: Async Celery Bridge via New Event Loop Per Task

Status: Accepted Date: 2026-04-29

Context

FastAPI routes are async (asyncio). Celery workers run in synchronous Python processes. MongoDB (Motor) and other async clients cannot be shared across asyncio event loop boundaries. Tasks need to call async services (Gemini, GCS, MongoDB, TTS) that only have async APIs.

Decision

Each Celery task creates a new asyncio.EventLoop via asyncio.new_event_loop() and runs its async implementation with loop.run_until_complete(task_impl()). The async implementation can freely use await with Motor, httpx, and other async clients. The event loop is closed in a finally block when the task completes.

Consequences

Benefits:

Async services work correctly inside Celery tasks
No shared mutable state between tasks
Each task is isolated — a failure does not corrupt another task's loop

Trade-offs:

Every task creates a new MongoDB connection (no connection pool reuse across tasks)
This is a known performance limitation; mitigation is connection pooling within the task's event loop lifetime
The pattern requires discipline: never await outside the task's own loop

Known violations: ingest_and_ai.py creates AsyncIOMotorClient(settings.mongodb_uri) directly instead of using a shared factory — should be extracted to a get_task_db() context manager.

Maintenance

Update triggers: If Celery adds native asyncio worker support, this ADR should be marked Deprecated and replaced.

1.6 KiB Raw Blame History