diff --git a/99 Daily/2026-04-30.md b/99 Daily/2026-04-30.md index 6f61c73..3b827f2 100644 --- a/99 Daily/2026-04-30.md +++ b/99 Daily/2026-04-30.md @@ -845,3 +845,9 @@ tags: [daily] - 21:40 (2min) | `video-accessibility` - **Asked:** Debug 403 Forbidden error on production queue stats API endpoint. - **Done:** Identified authentication issue with GET request to /video-accessibility/api/v1/admin/production/queue-stats endpoint. +- 21:47 | `video-accessibility` + - **Asked:** Investigated 403 Forbidden error on production queue-stats API endpoint. + - **Done:** Confirmed error was pre-existing and not caused by recent changes; ready to commit. +- 21:48 | `video-accessibility` + - **Asked:** The developer asked to fix projects not loading and expand language options. + - **Done:** Fixed projects loading with new backend endpoint and hook, expanded languages from 12 to 52 variants. diff --git a/wiki/_master-index.md b/wiki/_master-index.md index de337bb..7e81259 100644 --- a/wiki/_master-index.md +++ b/wiki/_master-index.md @@ -23,7 +23,7 @@ This 3-hop pattern works for hundreds of articles without vector search. | [[wiki/tech-patterns/_index\|tech-patterns/]] | Recurring tech stacks: FastAPI, React/Vite, Next.js, Azure AD, AI, Box, One2Edit, Redis/Celery, cost-tracker | 17 | | [[wiki/architecture/_index\|architecture/]] | Cross-cutting architectural patterns: Docker Compose, multi-agent AI, GCP timeout, RAG, hotfolder, optical-dev deploy, cost-tracker, new-project checklist, troubleshooting playbooks, ADR log, Cloud Run Jobs | 11 | | [[wiki/client-knowledge/_index\|client-knowledge/]] | Per-client notes for Ford, H&M, L'Oréal, Barclays, Ferrero, 3M | 6 | -| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 89 | +| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 93 | | [[wiki/connections/_index\|connections/]] | Cross-cutting insights linking 2+ concepts: FastAPI+Azure AD+Docker trinity, AI→cost-tracker, Apache+Vite basePath, GCP→REST polling, Box+hotfolder, Docker DNS+AdGuard, Celery prefork×faster_whisper memory stacking | 10 | | [[wiki/qa/_index\|qa/]] | Filed answers to queries (saved with `--file-back`) | 0 | | [[wiki/homelab/_index\|homelab/]] | Self-hosted infra: Proxmox install, IOMMU/PCI passthrough, hypervisor setup, budget builds, HP Elitedesk G3, Homarr API + Apps + Boards + Certificates + Integrations + Settings + Tasks + AdGuard + Clock + Docker Stats + Docker Integration + Download Client + Firewall + Proxmox Integration + Radarr + Readarr + Sonarr + Bookmarks + Calendar + Icons + App Widget + Weather + GitHub + Nextcloud + qBittorrent + RSS Feed + Speedtest Tracker + System Health Monitoring + System Resources + Services Map + Media Stack | 42 | diff --git a/wiki/concepts/_index.md b/wiki/concepts/_index.md index 1a63d02..d07b55c 100644 --- a/wiki/concepts/_index.md +++ b/wiki/concepts/_index.md @@ -104,5 +104,10 @@ | [[wiki/concepts/sudo-git-clone-root-ownership]] | `sudo git clone` makes all files root-owned — subsequent user `git pull` fails with Permission denied on .git/FETCH_HEAD; fix: chown -R | daily/2026-04-30.md | 2026-04-30 | | [[wiki/concepts/python-fastapi-module-level-singletons]] | `settings = Settings()` at module import level crashes pytest when env vars aren't set — guard with `@lru_cache` function or lazy `@property` | daily/2026-04-30.md | 2026-04-30 | +| [[wiki/concepts/react-query-enabled-falsy-value]] | `enabled: !!clientId` silently disables React Query when clientId is `""` — empty string is falsy; use explicit null check or separate hook | daily/2026-04-30.md | 2026-04-30 | +| [[wiki/concepts/mongodb-cross-collection-id-confusion]] | `find_one({"_id": client_id})` in wrong collection returns `None` silently — MongoDB has no FK constraints; email/notify code never runs | daily/2026-04-30.md | 2026-04-30 | +| [[wiki/concepts/browser-sequential-download-blocking]] | Sequential `window.open()` downloads only deliver the first file — browser popup blocker suppresses the rest; 400–500ms gap or fetch+blob fixes it | daily/2026-04-30.md | 2026-04-30 | +| [[wiki/concepts/mongodb-schema-validator-migration-verification]] | MongoDB `collMod` migration marked "applied" without actually running — verify with `getCollectionInfos`; fix with direct `collMod` + `validationLevel: moderate` | daily/2026-04-30.md | 2026-04-30 | + diff --git a/wiki/concepts/browser-sequential-download-blocking.md b/wiki/concepts/browser-sequential-download-blocking.md new file mode 100644 index 0000000..3d8e263 --- /dev/null +++ b/wiki/concepts/browser-sequential-download-blocking.md @@ -0,0 +1,95 @@ +--- +title: "Browser Sequential Download Blocking — window.open Needs 400ms Gap" +aliases: [browser-download-blocking, sequential-download, window-open-popup-blocker] +tags: [browser, javascript, frontend, download, popup-blocker, gotcha] +sources: + - "daily/2026-04-30.md" +created: 2026-04-30 +updated: 2026-04-30 +--- + +# Browser Sequential Download Blocking — `window.open` Needs 400ms Gap + +Browsers suppress sequential `window.open()` calls that happen too close together, treating them as a popup storm. When a "Download All" feature triggers multiple file downloads in a loop, only the first download succeeds — the rest are silently blocked by the browser's popup blocker. Adding a 400ms delay between each call resolves this. + +## Key Points + +- **Browsers allow only one `window.open()` per user gesture** — subsequent calls in the same synchronous or near-synchronous execution are treated as unsolicited popups and blocked +- **Only the first download succeeds** — there's no error thrown; the remaining files silently disappear +- **The fix:** add a 400ms (or longer) async gap between each download trigger — this allows the browser to process each "gesture" separately +- **`fetch` + `URL.createObjectURL` is the safer alternative** — avoids the popup blocker entirely by keeping the download in the same window context +- The exact threshold varies by browser (Chrome ~300ms, Firefox ~400ms, Safari ~500ms) — use 500ms for cross-browser safety + +## Details + +### The Problem + +```typescript +// ❌ BROKEN — only first download succeeds +async function downloadAll(fileUrls: string[]) { + for (const url of fileUrls) { + window.open(url, "_blank"); // ← 2nd, 3rd, etc. are silently blocked + } +} +``` + +### Fix 1: Async Gap Between Downloads + +```typescript +// ✅ Add delay between each window.open call +async function downloadAll(fileUrls: string[]) { + for (const url of fileUrls) { + window.open(url, "_blank"); + await new Promise(resolve => setTimeout(resolve, 500)); // 500ms gap + } +} +``` + +### Fix 2: Fetch + Blob URL (Popup-Blocker-Immune) + +```typescript +// ✅ More reliable — no popup blocker involvement +async function downloadFile(url: string, filename: string) { + const response = await fetch(url); + const blob = await response.blob(); + const blobUrl = URL.createObjectURL(blob); + + const anchor = document.createElement("a"); + anchor.href = blobUrl; + anchor.download = filename; + anchor.click(); + + URL.revokeObjectURL(blobUrl); // cleanup +} + +async function downloadAll(files: { url: string; name: string }[]) { + for (const file of files) { + await downloadFile(file.url, file.name); + await new Promise(resolve => setTimeout(resolve, 200)); // shorter gap needed + } +} +``` + +The `` approach doesn't trigger the popup blocker because it's a same-page navigation, not a new window. + +### When `window.open` Is Required + +For files hosted on a different origin (where `fetch` would be blocked by CORS), or for PDFs that need to open in a new tab, `window.open` may be unavoidable. In that case, the 500ms gap is the only option. + +### Diagnosing the Issue + +In Chrome DevTools: +- A blocked popup shows a small icon in the address bar ("Popup blocked") +- The browser console logs: `[blocked] Opening a URL that was denied by the popup policy` +- Or: `The window was not opened — popup blocker is active` + +If only the first of N downloads completes, popup blocking is almost certainly the cause. + +## Related Concepts + +- [[wiki/concepts/native-track-blob-url]] — related browser URL object pattern for VTT tracks +- [[wiki/tech-patterns/react-vite-typescript]] — React frontend patterns + +## Sources + +- [[daily/2026-04-30.md]] — video-accessibility Download All button: sequential `window.open` calls triggered popup blocker; only first file downloaded; fix was 400ms async gap between calls diff --git a/wiki/concepts/celery-prefork-pool-startup-memory.md b/wiki/concepts/celery-prefork-pool-startup-memory.md index 1d50c90..28db6b6 100644 --- a/wiki/concepts/celery-prefork-pool-startup-memory.md +++ b/wiki/concepts/celery-prefork-pool-startup-memory.md @@ -70,6 +70,10 @@ If a worker imports a model library at module level (e.g. `faster_whisper`, `tor - `docker stats` shows memory spike to container limit then drop (restart) - Tasks never start processing; queue builds up +### Real Incident (2026-04-30) + +`ffmpeg-worker` container set `CONCURRENCY=20` with ~120 MB per forked process. Total startup memory: **2.4 GB** — consumed before any task was processed. Container hit OOM limit and was killed by Docker within seconds of `docker compose up`. The pipeline stalled for **15 minutes** while the cause was invisible in application logs (no Python traceback, just container restart loop). Diagnosis: `docker stats` showed memory spike to limit then immediate drop, repeated every ~30 seconds. Fix: reduce `CONCURRENCY` using the formula `floor(container_memory_MB / per_worker_MB)`. + ## Related Concepts - [[wiki/concepts/faster-whisper-startup-memory]] — model loads at startup in each worker process diff --git a/wiki/concepts/mongodb-cross-collection-id-confusion.md b/wiki/concepts/mongodb-cross-collection-id-confusion.md new file mode 100644 index 0000000..53bc767 --- /dev/null +++ b/wiki/concepts/mongodb-cross-collection-id-confusion.md @@ -0,0 +1,95 @@ +--- +title: "MongoDB — Cross-Collection ObjectId Reference Confusion" +aliases: [mongodb-objectid-wrong-collection, mongodb-cross-collection-id, mongodb-silent-null-lookup] +tags: [mongodb, debugging, python, fastapi, gotcha, silent-failure] +sources: + - "daily/2026-04-30.md" +created: 2026-04-30 +updated: 2026-04-30 +--- + +# MongoDB — Cross-Collection ObjectId Reference Confusion + +When a document field stores a reference to collection A, using that value as a `_id` lookup in collection B silently returns `None` — MongoDB finds no matching document and returns nothing, causing the calling code to proceed with null data (silent failure, not an exception). + +## Key Points + +- **MongoDB returns `None` (not an error) for a valid ObjectId that doesn't exist in the queried collection** — there's no `404` or exception; `find_one` simply returns `None` +- **The failure is completely silent** if the calling code doesn't handle `None` explicitly, the function skips the operation without error +- **Common pattern:** a job stores `client_id` (reference to `db.clients`) but code accidentally looks it up in `db.users` — every lookup returns `None`, feature silently never works +- **Diagnosis:** add a `print/log` right after the `find_one` to confirm what was returned before assuming the downstream logic is the bug +- **Fix:** look up in the correct collection, or query the target collection using a field that references the ID (e.g., `db.users.find({"pm_client_ids": client_id})`) + +## Details + +### The Silent Failure Pattern + +```python +# job document: {"_id": ObjectId("abc..."), "client_id": ObjectId("xyz..."), ...} +# "client_id" is an ObjectId from db.clients collection + +# ❌ WRONG — looking up client_id in db.users where it doesn't exist +user = await db.users.find_one({"_id": client_id}) +# Returns None — ObjectId "xyz..." doesn't exist in users collection +# No error raised, user is None + +if user: + send_email(user["email"]) # ← never executes — email silently never sent +``` + +The function exits normally (no exception), logging shows it "ran", but the email was never sent. + +### The Fix: Query the Right Collection + +```python +# ✅ Option 1: look up in the correct collection first +client = await db.clients.find_one({"_id": client_id}) +if not client: + logger.warning(f"Client {client_id} not found") + return + +# Then find users who belong to this client +users = await db.users.find({"pm_client_ids": client_id}).to_list(None) +``` + +```python +# ✅ Option 2: query users by their reference to the client +users = await db.users.find({ + "pm_client_ids": client_id, + "role": {"$in": ["pm", "production"]} +}).to_list(None) + +# Fallback if no users matched +if not users: + users = await db.users.find({"role": "admin"}).to_list(None) +``` + +### Why This Happens + +MongoDB ObjectIds are just 12-byte identifiers with no type information attached — they don't know which collection they "belong to". An ObjectId `xyz...` from `db.clients` is structurally identical to an ObjectId `xyz...` from `db.users`. Python code that passes one where the other is expected will not get a type error; it will get `None` from the database. + +In relational databases (Postgres), a foreign key constraint would catch this at the DB level. MongoDB has no such constraint. + +### Detection + +Add an assertion or explicit check right after `find_one`: + +```python +user = await db.users.find_one({"_id": client_id}) +assert user is not None, f"BUG: expected user for {client_id}, got None — check collection" +``` + +Or log immediately: + +```python +logger.debug(f"Looking up user by _id={client_id}: found={user is not None}") +``` + +## Related Concepts + +- [[wiki/concepts/multitenant-fail-open-authz]] — another silent null return pattern where `None` leads to fail-open behavior +- [[wiki/concepts/pydantic-v2-alias-id-gotcha]] — related Pydantic/MongoDB ID mapping pitfalls + +## Sources + +- [[daily/2026-04-30.md]] — `notify.py` in video-accessibility: `db.users.find_one({"_id": client_id})` returned `None` because `client_id` references `db.clients`, not users; fix was `db.users.find({"pm_client_ids": client_id})` with fallback to admin users diff --git a/wiki/concepts/mongodb-schema-validator-migration-verification.md b/wiki/concepts/mongodb-schema-validator-migration-verification.md new file mode 100644 index 0000000..fa9e2f6 --- /dev/null +++ b/wiki/concepts/mongodb-schema-validator-migration-verification.md @@ -0,0 +1,121 @@ +--- +title: "MongoDB — Schema Validator Migrations Can Be Silently Skipped" +aliases: [mongodb-schema-validator, mongodb-collmod-migration, mongodb-jsonschema-verification] +tags: [mongodb, migrations, debugging, python, fastapi, gotcha] +sources: + - "daily/2026-04-30.md" +created: 2026-04-30 +updated: 2026-04-30 +--- + +# MongoDB — Schema Validator Migrations Can Be Silently Skipped + +MongoDB migration systems (like `migrate.py` pattern) record migrations as "applied" in a migrations collection. If the `collMod` command inside a migration fails silently (e.g., network hiccup, auth issue, wrong database) or was never executed, the migration is still marked applied — and `migrate up` returns "No migrations to apply" on subsequent runs. The validator is never updated, and writes are rejected with validation errors. + +## Key Points + +- **Migrations marked "applied" does NOT mean the `collMod` ran** — the migration tracking record and the actual schema change are independent +- **Symptom:** worker code writes a new enum value to a field (`processing_failed`, `tts_generating`), MongoDB rejects with a validation error, service crashes or enters a retry loop +- **Diagnosis:** verify the actual validator with `db.getCollectionInfos({name:"collection_name"})[0].options.validator` +- **Fix:** run the `collMod` directly in the MongoDB shell (or via eval) — no need to manipulate migration history +- **Prevention:** always add a post-migration assertion that reads back the validator and confirms the new value is present + +## Details + +### The Silent Migration Failure + +```python +# migrations/2026-04-29-000000_add_processing_failed_status.py + +async def up(db): + await db.command({ + "collMod": "jobs", + "validator": { + "$jsonSchema": { + "properties": { + "status": { + "enum": ["created", "processing", "completed", "processing_failed"] + } + } + } + } + }) + # If this fails silently → migration table records "applied" anyway + # Next run: "No migrations to apply" — but the validator was never updated +``` + +### Verifying the Current Validator + +```javascript +// MongoDB shell — check what's actually in the validator +db.getCollectionInfos({name: "jobs"})[0].options.validator + +// Or via Python +info = await db.command("listCollections", filter={"name": "jobs"}) +validator = info["cursor"]["firstBatch"][0]["options"].get("validator") +print(validator) +``` + +Look for the `enum` list under the relevant field. If `processing_failed` is absent, the migration didn't run. + +### Direct Fix via collMod + +When the migration is already marked applied and `migrate up` won't rerun it: + +```javascript +// MongoDB shell — run directly +db.runCommand({ + collMod: "jobs", + validator: { + "$jsonSchema": { + "bsonType": "object", + "properties": { + "status": { + "bsonType": "string", + "enum": ["created", "processing", "completed", "processing_failed", "tts_generating"] + } + } + } + }, + validationLevel: "moderate" +}) +``` + +`validationLevel: "moderate"` allows existing documents that don't conform to remain readable — only new inserts/updates are validated. + +Or via Python inline eval: + +```python +await db.command( + "collMod", "jobs", + validator={"$jsonSchema": {...}}, + validationLevel="moderate" +) +``` + +### Prevention: Post-Migration Assertion + +```python +async def up(db): + await db.command({"collMod": "jobs", "validator": {...}}) + + # VERIFY the change actually took effect + info = (await db.list_collections(filter={"name": "jobs"}).to_list(1))[0] + validator = info.get("options", {}).get("validator", {}) + schema = validator.get("$jsonSchema", {}) + status_enum = schema.get("properties", {}).get("status", {}).get("enum", []) + assert "processing_failed" in status_enum, "Migration collMod did not apply!" +``` + +### Real Incident (2026-04-30) + +`video-accessibility` workers tried to write `status: "processing_failed"` to a job document. MongoDB `$jsonSchema` validator rejected it (enum only had `created`, `processing`, `completed`). The migration `2026-04-29-000000_add_processing_failed_status_and_indexes.py` was recorded as applied in the migration tracking collection, but the `collMod` never ran. Worker entered a retry loop (same validation error every attempt). Fix was running `collMod` directly with `--eval` in the MongoDB shell. + +## Related Concepts + +- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — when a deterministic error (like validation rejection) causes Celery retry loops, Redis must also be flushed +- [[wiki/concepts/mongodb-enum-deserialization]] — related MongoDB enum handling on the read side + +## Sources + +- [[daily/2026-04-30.md]] — `processing_failed` status write rejected by MongoDB validator; migration was marked applied but `collMod` never ran; fixed with direct `collMod` via shell eval; `getCollectionInfos` used to verify diff --git a/wiki/concepts/react-query-enabled-falsy-value.md b/wiki/concepts/react-query-enabled-falsy-value.md new file mode 100644 index 0000000..98b0674 --- /dev/null +++ b/wiki/concepts/react-query-enabled-falsy-value.md @@ -0,0 +1,88 @@ +--- +title: "React Query — enabled: !!value Silent Skip on Empty String" +aliases: [react-query-enabled, react-query-silent-skip, usequery-disabled] +tags: [react, react-query, frontend, debugging, gotcha, javascript] +sources: + - "daily/2026-04-30.md" +created: 2026-04-30 +updated: 2026-04-30 +--- + +# React Query — `enabled: !!value` Silent Skip on Empty String + +When a React Query hook uses `enabled: !!someId` as a condition to fire the request, passing an empty string `""` as `someId` silently disables the query — `!!""` is `false`. The query never fires, the loading state resolves immediately with `undefined` data, and the UI shows nothing without any error or console warning. + +## Key Points + +- **`!!""` is `false`** — empty string is falsy in JavaScript; React Query treats it as "disabled" +- **Symptom:** component renders, no network request is made, data is `undefined` — looks like an API error but there is no request at all +- **Diagnosis:** in React Query DevTools or browser network tab — if there's no request for a query that should have fired, check `enabled` condition for falsy values +- **Fix option A:** use a separate hook or endpoint that doesn't require the ID (e.g., `GET /clients/all-projects` that returns all projects without a client filter) +- **Fix option B:** use `enabled: someId !== null && someId !== undefined` instead of `enabled: !!someId` to allow empty-string IDs + +## Details + +### The Failure Pattern + +```typescript +// useProjects hook — requires clientId +const { data: projects } = useQuery( + ["projects", clientId], + () => fetchProjects(clientId), + { enabled: !!clientId } // ← silently disabled when clientId === "" +); + +// In a form where the user hasn't selected a client yet: +const [clientId, setClientId] = useState(""); // empty string default +// → query never fires, projects is undefined, dropdown is empty +``` + +### Fix A: Separate "All Projects" Endpoint + +When the UI needs ALL projects regardless of client selection, create a separate endpoint and hook that doesn't require a `clientId`: + +```typescript +// New hook — no clientId required +export function useAllProjects() { + return useQuery(["projects", "all"], fetchAllProjects); // always enabled +} + +// New backend endpoint +// GET /clients/all-projects → returns all projects user has access to +``` + +This avoids changing the `enabled` logic and makes the "fetch all" intent explicit. + +### Fix B: Explicit Null Check + +When the ID is legitimately optional but not always empty string: + +```typescript +const { data: projects } = useQuery( + ["projects", clientId], + () => fetchProjects(clientId), + { enabled: clientId !== null && clientId !== undefined } + // now "" (empty string) is allowed and the query fires +); +``` + +### Common Falsy Values to Watch For in `enabled` + +| Value | `!!value` | Often means | +|-------|-----------|-------------| +| `""` | `false` | Default state, unselected dropdown | +| `0` | `false` | First item in a zero-indexed list | +| `null` | `false` | Not yet loaded | +| `undefined` | `false` | Not yet loaded | +| `"0"` | `true` (string!) | String zero — may be surprising | + +When in doubt, use `enabled: value !== null && value !== undefined` rather than `enabled: !!value`. + +## Related Concepts + +- [[wiki/concepts/zustand-async-hydration]] — another silent timing bug in React where state isn't ready when components mount +- [[wiki/tech-patterns/react-vite-typescript]] — React patterns in Oliver projects + +## Sources + +- [[daily/2026-04-30.md]] — Brief form: projects dropdown empty because `useProjects('')` was disabled by `enabled: !!clientId`; fixed with `useAllProjects()` hook + `GET /clients/all-projects` endpoint diff --git a/wiki/log.md b/wiki/log.md index c361579..beb9a6e 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -1,6 +1,12 @@ # Build Log +## [2026-04-30T23:59:00+01:00] compile | 2026-04-30.md (pass 3) +- Source: daily/2026-04-30.md +- Articles created: [[wiki/concepts/react-query-enabled-falsy-value]], [[wiki/concepts/mongodb-cross-collection-id-confusion]], [[wiki/concepts/browser-sequential-download-blocking]], [[wiki/concepts/mongodb-schema-validator-migration-verification]] +- Articles updated: [[wiki/concepts/celery-prefork-pool-startup-memory]] (added Real Incident section: CONCURRENCY=20, ~120 MB/process, 2.4 GB total, OOM before first task, 15-minute stall) +- Index updates: [[wiki/concepts/_index]] (89→93); [[wiki/_master-index]] (concepts 89→93) + ## [2026-04-30T23:30:00+01:00] compile | 2026-04-30.md (pass 2) - Source: daily/2026-04-30.md - Articles created: [[wiki/concepts/celery-prefork-pool-startup-memory]], [[wiki/concepts/sudo-git-clone-root-ownership]], [[wiki/concepts/python-fastapi-module-level-singletons]], [[wiki/connections/celery-prefork-faster-whisper-memory-stacking]]