vault backup: 2026-05-01 09:38:54

99 Daily/2026-04-30.md

@@ -845,3 +845,9 @@ tags: [daily]
 - 21:40 (2min) | `video-accessibility`
   - **Asked:** Debug 403 Forbidden error on production queue stats API endpoint.
   - **Done:** Identified authentication issue with GET request to /video-accessibility/api/v1/admin/production/queue-stats endpoint.
+- 21:47 | `video-accessibility`
+  - **Asked:** Investigated 403 Forbidden error on production queue-stats API endpoint.
+  - **Done:** Confirmed error was pre-existing and not caused by recent changes; ready to commit.
+- 21:48 | `video-accessibility`
+  - **Asked:** The developer asked to fix projects not loading and expand language options.
+  - **Done:** Fixed projects loading with new backend endpoint and hook, expanded languages from 12 to 52 variants.

wiki/_master-index.md

@@ -23,7 +23,7 @@ This 3-hop pattern works for hundreds of articles without vector search.
 | [[wiki/tech-patterns/_index\|tech-patterns/]] | Recurring tech stacks: FastAPI, React/Vite, Next.js, Azure AD, AI, Box, One2Edit, Redis/Celery, cost-tracker | 17 |
 | [[wiki/architecture/_index\|architecture/]] | Cross-cutting architectural patterns: Docker Compose, multi-agent AI, GCP timeout, RAG, hotfolder, optical-dev deploy, cost-tracker, new-project checklist, troubleshooting playbooks, ADR log, Cloud Run Jobs | 11 |
 | [[wiki/client-knowledge/_index\|client-knowledge/]] | Per-client notes for Ford, H&M, L'Oréal, Barclays, Ferrero, 3M | 6 |
-| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 89 |
+| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 93 |
 | [[wiki/connections/_index\|connections/]] | Cross-cutting insights linking 2+ concepts: FastAPI+Azure AD+Docker trinity, AI→cost-tracker, Apache+Vite basePath, GCP→REST polling, Box+hotfolder, Docker DNS+AdGuard, Celery prefork×faster_whisper memory stacking | 10 |
 | [[wiki/qa/_index\|qa/]] | Filed answers to queries (saved with `--file-back`) | 0 |
 | [[wiki/homelab/_index\|homelab/]] | Self-hosted infra: Proxmox install, IOMMU/PCI passthrough, hypervisor setup, budget builds, HP Elitedesk G3, Homarr API + Apps + Boards + Certificates + Integrations + Settings + Tasks + AdGuard + Clock + Docker Stats + Docker Integration + Download Client + Firewall + Proxmox Integration + Radarr + Readarr + Sonarr + Bookmarks + Calendar + Icons + App Widget + Weather + GitHub + Nextcloud + qBittorrent + RSS Feed + Speedtest Tracker + System Health Monitoring + System Resources + Services Map + Media Stack | 42 |

wiki/concepts/_index.md

@@ -104,5 +104,10 @@
 | [[wiki/concepts/sudo-git-clone-root-ownership]] | `sudo git clone` makes all files root-owned — subsequent user `git pull` fails with Permission denied on .git/FETCH_HEAD; fix: chown -R | daily/2026-04-30.md | 2026-04-30 |
 | [[wiki/concepts/python-fastapi-module-level-singletons]] | `settings = Settings()` at module import level crashes pytest when env vars aren't set — guard with `@lru_cache` function or lazy `@property` | daily/2026-04-30.md | 2026-04-30 |
 
+| [[wiki/concepts/react-query-enabled-falsy-value]] | `enabled: !!clientId` silently disables React Query when clientId is `""` — empty string is falsy; use explicit null check or separate hook | daily/2026-04-30.md | 2026-04-30 |
+| [[wiki/concepts/mongodb-cross-collection-id-confusion]] | `find_one({"_id": client_id})` in wrong collection returns `None` silently — MongoDB has no FK constraints; email/notify code never runs | daily/2026-04-30.md | 2026-04-30 |
+| [[wiki/concepts/browser-sequential-download-blocking]] | Sequential `window.open()` downloads only deliver the first file — browser popup blocker suppresses the rest; 400–500ms gap or fetch+blob fixes it | daily/2026-04-30.md | 2026-04-30 |
+| [[wiki/concepts/mongodb-schema-validator-migration-verification]] | MongoDB `collMod` migration marked "applied" without actually running — verify with `getCollectionInfos`; fix with direct `collMod` + `validationLevel: moderate` | daily/2026-04-30.md | 2026-04-30 |
+
 <!-- Articles added automatically by compile.py -->
 <!-- Format: | [[concepts/slug]] | One-line summary | daily/YYYY-MM-DD.md | date | -->

wiki/concepts/browser-sequential-download-blocking.md

@@ -0,0 +1,95 @@
+---
+title: "Browser Sequential Download Blocking — window.open Needs 400ms Gap"
+aliases: [browser-download-blocking, sequential-download, window-open-popup-blocker]
+tags: [browser, javascript, frontend, download, popup-blocker, gotcha]
+sources:
+  - "daily/2026-04-30.md"
+created: 2026-04-30
+updated: 2026-04-30
+---
+
+# Browser Sequential Download Blocking — `window.open` Needs 400ms Gap
+
+Browsers suppress sequential `window.open()` calls that happen too close together, treating them as a popup storm. When a "Download All" feature triggers multiple file downloads in a loop, only the first download succeeds — the rest are silently blocked by the browser's popup blocker. Adding a 400ms delay between each call resolves this.
+
+## Key Points
+
+- **Browsers allow only one `window.open()` per user gesture** — subsequent calls in the same synchronous or near-synchronous execution are treated as unsolicited popups and blocked
+- **Only the first download succeeds** — there's no error thrown; the remaining files silently disappear
+- **The fix:** add a 400ms (or longer) async gap between each download trigger — this allows the browser to process each "gesture" separately
+- **`fetch` + `URL.createObjectURL` is the safer alternative** — avoids the popup blocker entirely by keeping the download in the same window context
+- The exact threshold varies by browser (Chrome ~300ms, Firefox ~400ms, Safari ~500ms) — use 500ms for cross-browser safety
+
+## Details
+
+### The Problem
+
+```typescript
+// ❌ BROKEN — only first download succeeds
+async function downloadAll(fileUrls: string[]) {
+  for (const url of fileUrls) {
+    window.open(url, "_blank");  // ← 2nd, 3rd, etc. are silently blocked
+  }
+}
+```
+
+### Fix 1: Async Gap Between Downloads
+
+```typescript
+// ✅ Add delay between each window.open call
+async function downloadAll(fileUrls: string[]) {
+  for (const url of fileUrls) {
+    window.open(url, "_blank");
+    await new Promise(resolve => setTimeout(resolve, 500));  // 500ms gap
+  }
+}
+```
+
+### Fix 2: Fetch + Blob URL (Popup-Blocker-Immune)
+
+```typescript
+// ✅ More reliable — no popup blocker involvement
+async function downloadFile(url: string, filename: string) {
+  const response = await fetch(url);
+  const blob = await response.blob();
+  const blobUrl = URL.createObjectURL(blob);
+
+  const anchor = document.createElement("a");
+  anchor.href = blobUrl;
+  anchor.download = filename;
+  anchor.click();
+
+  URL.revokeObjectURL(blobUrl);  // cleanup
+}
+
+async function downloadAll(files: { url: string; name: string }[]) {
+  for (const file of files) {
+    await downloadFile(file.url, file.name);
+    await new Promise(resolve => setTimeout(resolve, 200));  // shorter gap needed
+  }
+}
+```
+
+The `<a download>` approach doesn't trigger the popup blocker because it's a same-page navigation, not a new window.
+
+### When `window.open` Is Required
+
+For files hosted on a different origin (where `fetch` would be blocked by CORS), or for PDFs that need to open in a new tab, `window.open` may be unavoidable. In that case, the 500ms gap is the only option.
+
+### Diagnosing the Issue
+
+In Chrome DevTools:
+- A blocked popup shows a small icon in the address bar ("Popup blocked")
+- The browser console logs: `[blocked] Opening a URL that was denied by the popup policy`
+- Or: `The window was not opened — popup blocker is active`
+
+If only the first of N downloads completes, popup blocking is almost certainly the cause.
+
+## Related Concepts
+
+- [[wiki/concepts/native-track-blob-url]] — related browser URL object pattern for VTT tracks
+- [[wiki/tech-patterns/react-vite-typescript]] — React frontend patterns
+
+## Sources
+
+- [[daily/2026-04-30.md]] — video-accessibility Download All button: sequential `window.open` calls triggered popup blocker; only first file downloaded; fix was 400ms async gap between calls

wiki/concepts/celery-prefork-pool-startup-memory.md

@@ -70,6 +70,10 @@ If a worker imports a model library at module level (e.g. `faster_whisper`, `tor
 - `docker stats` shows memory spike to container limit then drop (restart)
 - Tasks never start processing; queue builds up
 
+### Real Incident (2026-04-30)
+
+`ffmpeg-worker` container set `CONCURRENCY=20` with ~120 MB per forked process. Total startup memory: **2.4 GB** — consumed before any task was processed. Container hit OOM limit and was killed by Docker within seconds of `docker compose up`. The pipeline stalled for **15 minutes** while the cause was invisible in application logs (no Python traceback, just container restart loop). Diagnosis: `docker stats` showed memory spike to limit then immediate drop, repeated every ~30 seconds. Fix: reduce `CONCURRENCY` using the formula `floor(container_memory_MB / per_worker_MB)`.
+
 ## Related Concepts
 
 - [[wiki/concepts/faster-whisper-startup-memory]] — model loads at startup in each worker process

wiki/concepts/mongodb-cross-collection-id-confusion.md

@@ -0,0 +1,95 @@
+---
+title: "MongoDB — Cross-Collection ObjectId Reference Confusion"
+aliases: [mongodb-objectid-wrong-collection, mongodb-cross-collection-id, mongodb-silent-null-lookup]
+tags: [mongodb, debugging, python, fastapi, gotcha, silent-failure]
+sources:
+  - "daily/2026-04-30.md"
+created: 2026-04-30
+updated: 2026-04-30
+---
+
+# MongoDB — Cross-Collection ObjectId Reference Confusion
+
+When a document field stores a reference to collection A, using that value as a `_id` lookup in collection B silently returns `None` — MongoDB finds no matching document and returns nothing, causing the calling code to proceed with null data (silent failure, not an exception).
+
+## Key Points
+
+- **MongoDB returns `None` (not an error) for a valid ObjectId that doesn't exist in the queried collection** — there's no `404` or exception; `find_one` simply returns `None`
+- **The failure is completely silent** if the calling code doesn't handle `None` explicitly, the function skips the operation without error
+- **Common pattern:** a job stores `client_id` (reference to `db.clients`) but code accidentally looks it up in `db.users` — every lookup returns `None`, feature silently never works
+- **Diagnosis:** add a `print/log` right after the `find_one` to confirm what was returned before assuming the downstream logic is the bug
+- **Fix:** look up in the correct collection, or query the target collection using a field that references the ID (e.g., `db.users.find({"pm_client_ids": client_id})`)
+
+## Details
+
+### The Silent Failure Pattern
+
+```python
+# job document: {"_id": ObjectId("abc..."), "client_id": ObjectId("xyz..."), ...}
+# "client_id" is an ObjectId from db.clients collection
+
+# ❌ WRONG — looking up client_id in db.users where it doesn't exist
+user = await db.users.find_one({"_id": client_id})
+# Returns None — ObjectId "xyz..." doesn't exist in users collection
+# No error raised, user is None
+
+if user:
+    send_email(user["email"])  # ← never executes — email silently never sent
+```
+
+The function exits normally (no exception), logging shows it "ran", but the email was never sent.
+
+### The Fix: Query the Right Collection
+
+```python
+# ✅ Option 1: look up in the correct collection first
+client = await db.clients.find_one({"_id": client_id})
+if not client:
+    logger.warning(f"Client {client_id} not found")
+    return
+
+# Then find users who belong to this client
+users = await db.users.find({"pm_client_ids": client_id}).to_list(None)
+```
+
+```python
+# ✅ Option 2: query users by their reference to the client
+users = await db.users.find({
+    "pm_client_ids": client_id,
+    "role": {"$in": ["pm", "production"]}
+}).to_list(None)
+
+# Fallback if no users matched
+if not users:
+    users = await db.users.find({"role": "admin"}).to_list(None)
+```
+
+### Why This Happens
+
+MongoDB ObjectIds are just 12-byte identifiers with no type information attached — they don't know which collection they "belong to". An ObjectId `xyz...` from `db.clients` is structurally identical to an ObjectId `xyz...` from `db.users`. Python code that passes one where the other is expected will not get a type error; it will get `None` from the database.
+
+In relational databases (Postgres), a foreign key constraint would catch this at the DB level. MongoDB has no such constraint.
+
+### Detection
+
+Add an assertion or explicit check right after `find_one`:
+
+```python
+user = await db.users.find_one({"_id": client_id})
+assert user is not None, f"BUG: expected user for {client_id}, got None — check collection"
+```
+
+Or log immediately:
+
+```python
+logger.debug(f"Looking up user by _id={client_id}: found={user is not None}")
+```
+
+## Related Concepts
+
+- [[wiki/concepts/multitenant-fail-open-authz]] — another silent null return pattern where `None` leads to fail-open behavior
+- [[wiki/concepts/pydantic-v2-alias-id-gotcha]] — related Pydantic/MongoDB ID mapping pitfalls
+
+## Sources
+
+- [[daily/2026-04-30.md]] — `notify.py` in video-accessibility: `db.users.find_one({"_id": client_id})` returned `None` because `client_id` references `db.clients`, not users; fix was `db.users.find({"pm_client_ids": client_id})` with fallback to admin users

wiki/concepts/mongodb-schema-validator-migration-verification.md

@@ -0,0 +1,121 @@
+---
+title: "MongoDB — Schema Validator Migrations Can Be Silently Skipped"
+aliases: [mongodb-schema-validator, mongodb-collmod-migration, mongodb-jsonschema-verification]
+tags: [mongodb, migrations, debugging, python, fastapi, gotcha]
+sources:
+  - "daily/2026-04-30.md"
+created: 2026-04-30
+updated: 2026-04-30
+---
+
+# MongoDB — Schema Validator Migrations Can Be Silently Skipped
+
+MongoDB migration systems (like `migrate.py` pattern) record migrations as "applied" in a migrations collection. If the `collMod` command inside a migration fails silently (e.g., network hiccup, auth issue, wrong database) or was never executed, the migration is still marked applied — and `migrate up` returns "No migrations to apply" on subsequent runs. The validator is never updated, and writes are rejected with validation errors.
+
+## Key Points
+
+- **Migrations marked "applied" does NOT mean the `collMod` ran** — the migration tracking record and the actual schema change are independent
+- **Symptom:** worker code writes a new enum value to a field (`processing_failed`, `tts_generating`), MongoDB rejects with a validation error, service crashes or enters a retry loop
+- **Diagnosis:** verify the actual validator with `db.getCollectionInfos({name:"collection_name"})[0].options.validator`
+- **Fix:** run the `collMod` directly in the MongoDB shell (or via eval) — no need to manipulate migration history
+- **Prevention:** always add a post-migration assertion that reads back the validator and confirms the new value is present
+
+## Details
+
+### The Silent Migration Failure
+
+```python
+# migrations/2026-04-29-000000_add_processing_failed_status.py
+
+async def up(db):
+    await db.command({
+        "collMod": "jobs",
+        "validator": {
+            "$jsonSchema": {
+                "properties": {
+                    "status": {
+                        "enum": ["created", "processing", "completed", "processing_failed"]
+                    }
+                }
+            }
+        }
+    })
+    # If this fails silently → migration table records "applied" anyway
+    # Next run: "No migrations to apply" — but the validator was never updated
+```
+
+### Verifying the Current Validator
+
+```javascript
+// MongoDB shell — check what's actually in the validator
+db.getCollectionInfos({name: "jobs"})[0].options.validator
+
+// Or via Python
+info = await db.command("listCollections", filter={"name": "jobs"})
+validator = info["cursor"]["firstBatch"][0]["options"].get("validator")
+print(validator)
+```
+
+Look for the `enum` list under the relevant field. If `processing_failed` is absent, the migration didn't run.
+
+### Direct Fix via collMod
+
+When the migration is already marked applied and `migrate up` won't rerun it:
+
+```javascript
+// MongoDB shell — run directly
+db.runCommand({
+    collMod: "jobs",
+    validator: {
+        "$jsonSchema": {
+            "bsonType": "object",
+            "properties": {
+                "status": {
+                    "bsonType": "string",
+                    "enum": ["created", "processing", "completed", "processing_failed", "tts_generating"]
+                }
+            }
+        }
+    },
+    validationLevel: "moderate"
+})
+```
+
+`validationLevel: "moderate"` allows existing documents that don't conform to remain readable — only new inserts/updates are validated.
+
+Or via Python inline eval:
+
+```python
+await db.command(
+    "collMod", "jobs",
+    validator={"$jsonSchema": {...}},
+    validationLevel="moderate"
+)
+```
+
+### Prevention: Post-Migration Assertion
+
+```python
+async def up(db):
+    await db.command({"collMod": "jobs", "validator": {...}})
+
+    # VERIFY the change actually took effect
+    info = (await db.list_collections(filter={"name": "jobs"}).to_list(1))[0]
+    validator = info.get("options", {}).get("validator", {})
+    schema = validator.get("$jsonSchema", {})
+    status_enum = schema.get("properties", {}).get("status", {}).get("enum", [])
+    assert "processing_failed" in status_enum, "Migration collMod did not apply!"
+```
+
+### Real Incident (2026-04-30)
+
+`video-accessibility` workers tried to write `status: "processing_failed"` to a job document. MongoDB `$jsonSchema` validator rejected it (enum only had `created`, `processing`, `completed`). The migration `2026-04-29-000000_add_processing_failed_status_and_indexes.py` was recorded as applied in the migration tracking collection, but the `collMod` never ran. Worker entered a retry loop (same validation error every attempt). Fix was running `collMod` directly with `--eval` in the MongoDB shell.
+
+## Related Concepts
+
+- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — when a deterministic error (like validation rejection) causes Celery retry loops, Redis must also be flushed
+- [[wiki/concepts/mongodb-enum-deserialization]] — related MongoDB enum handling on the read side
+
+## Sources
+
+- [[daily/2026-04-30.md]] — `processing_failed` status write rejected by MongoDB validator; migration was marked applied but `collMod` never ran; fixed with direct `collMod` via shell eval; `getCollectionInfos` used to verify

wiki/concepts/react-query-enabled-falsy-value.md

@@ -0,0 +1,88 @@
+---
+title: "React Query — enabled: !!value Silent Skip on Empty String"
+aliases: [react-query-enabled, react-query-silent-skip, usequery-disabled]
+tags: [react, react-query, frontend, debugging, gotcha, javascript]
+sources:
+  - "daily/2026-04-30.md"
+created: 2026-04-30
+updated: 2026-04-30
+---
+
+# React Query — `enabled: !!value` Silent Skip on Empty String
+
+When a React Query hook uses `enabled: !!someId` as a condition to fire the request, passing an empty string `""` as `someId` silently disables the query — `!!""` is `false`. The query never fires, the loading state resolves immediately with `undefined` data, and the UI shows nothing without any error or console warning.
+
+## Key Points
+
+- **`!!""` is `false`** — empty string is falsy in JavaScript; React Query treats it as "disabled"
+- **Symptom:** component renders, no network request is made, data is `undefined` — looks like an API error but there is no request at all
+- **Diagnosis:** in React Query DevTools or browser network tab — if there's no request for a query that should have fired, check `enabled` condition for falsy values
+- **Fix option A:** use a separate hook or endpoint that doesn't require the ID (e.g., `GET /clients/all-projects` that returns all projects without a client filter)
+- **Fix option B:** use `enabled: someId !== null && someId !== undefined` instead of `enabled: !!someId` to allow empty-string IDs
+
+## Details
+
+### The Failure Pattern
+
+```typescript
+// useProjects hook — requires clientId
+const { data: projects } = useQuery(
+  ["projects", clientId],
+  () => fetchProjects(clientId),
+  { enabled: !!clientId }   // ← silently disabled when clientId === ""
+);
+
+// In a form where the user hasn't selected a client yet:
+const [clientId, setClientId] = useState("");  // empty string default
+// → query never fires, projects is undefined, dropdown is empty
+```
+
+### Fix A: Separate "All Projects" Endpoint
+
+When the UI needs ALL projects regardless of client selection, create a separate endpoint and hook that doesn't require a `clientId`:
+
+```typescript
+// New hook — no clientId required
+export function useAllProjects() {
+  return useQuery(["projects", "all"], fetchAllProjects);  // always enabled
+}
+
+// New backend endpoint
+// GET /clients/all-projects → returns all projects user has access to
+```
+
+This avoids changing the `enabled` logic and makes the "fetch all" intent explicit.
+
+### Fix B: Explicit Null Check
+
+When the ID is legitimately optional but not always empty string:
+
+```typescript
+const { data: projects } = useQuery(
+  ["projects", clientId],
+  () => fetchProjects(clientId),
+  { enabled: clientId !== null && clientId !== undefined }
+  // now "" (empty string) is allowed and the query fires
+);
+```
+
+### Common Falsy Values to Watch For in `enabled`
+
+| Value | `!!value` | Often means |
+|-------|-----------|-------------|
+| `""` | `false` | Default state, unselected dropdown |
+| `0` | `false` | First item in a zero-indexed list |
+| `null` | `false` | Not yet loaded |
+| `undefined` | `false` | Not yet loaded |
+| `"0"` | `true` (string!) | String zero — may be surprising |
+
+When in doubt, use `enabled: value !== null && value !== undefined` rather than `enabled: !!value`.
+
+## Related Concepts
+
+- [[wiki/concepts/zustand-async-hydration]] — another silent timing bug in React where state isn't ready when components mount
+- [[wiki/tech-patterns/react-vite-typescript]] — React patterns in Oliver projects
+
+## Sources
+
+- [[daily/2026-04-30.md]] — Brief form: projects dropdown empty because `useProjects('')` was disabled by `enabled: !!clientId`; fixed with `useAllProjects()` hook + `GET /clients/all-projects` endpoint

wiki/log.md

@@ -1,6 +1,12 @@
 
 # Build Log
 
+## [2026-04-30T23:59:00+01:00] compile | 2026-04-30.md (pass 3)
+- Source: daily/2026-04-30.md
+- Articles created: [[wiki/concepts/react-query-enabled-falsy-value]], [[wiki/concepts/mongodb-cross-collection-id-confusion]], [[wiki/concepts/browser-sequential-download-blocking]], [[wiki/concepts/mongodb-schema-validator-migration-verification]]
+- Articles updated: [[wiki/concepts/celery-prefork-pool-startup-memory]] (added Real Incident section: CONCURRENCY=20, ~120 MB/process, 2.4 GB total, OOM before first task, 15-minute stall)
+- Index updates: [[wiki/concepts/_index]] (89→93); [[wiki/_master-index]] (concepts 89→93)
+
 ## [2026-04-30T23:30:00+01:00] compile | 2026-04-30.md (pass 2)
 - Source: daily/2026-04-30.md
 - Articles created: [[wiki/concepts/celery-prefork-pool-startup-memory]], [[wiki/concepts/sudo-git-clone-root-ownership]], [[wiki/concepts/python-fastapi-module-level-singletons]], [[wiki/connections/celery-prefork-faster-whisper-memory-stacking]]