diff --git a/wiki/_master-index.md b/wiki/_master-index.md index 7beabd2..f1de9ee 100644 --- a/wiki/_master-index.md +++ b/wiki/_master-index.md @@ -23,7 +23,7 @@ This 3-hop pattern works for hundreds of articles without vector search. | [[wiki/tech-patterns/_index\|tech-patterns/]] | Recurring tech stacks: FastAPI, React/Vite, Next.js, Azure AD, AI, Box, One2Edit, Redis/Celery, cost-tracker, OMG API | 28 | | [[wiki/architecture/_index\|architecture/]] | Cross-cutting architectural patterns: Docker Compose, multi-agent AI, GCP timeout, RAG, hotfolder, optical-dev deploy, cost-tracker, new-project checklist, troubleshooting playbooks, ADR log, Cloud Run Jobs | 11 | | [[wiki/client-knowledge/_index\|client-knowledge/]] | Per-client notes for Ford, H&M, L'Oréal, Barclays, Ferrero, 3M, BAIC | 7 | -| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 166 | +| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 174 | | [[wiki/connections/_index\|connections/]] | Cross-cutting insights linking 2+ concepts: FastAPI+Azure AD+Docker trinity, AI→cost-tracker, Apache+Vite basePath, GCP→REST polling, Box+hotfolder, Docker DNS+AdGuard, Celery prefork×faster_whisper memory stacking | 10 | | [[wiki/qa/_index\|qa/]] | Filed answers to queries (saved with `--file-back`) | 0 | | [[wiki/homelab/_index\|homelab/]] | Self-hosted infra: Proxmox install, IOMMU/PCI passthrough, hypervisor setup, budget builds, HP Elitedesk G3, Homarr API + Apps + Boards + Certificates + Integrations + Settings + Tasks + AdGuard + Clock + Docker Stats + Docker Integration + Download Client + Firewall + Proxmox Integration + Radarr + Readarr + Sonarr + Bookmarks + Calendar + Icons + App Widget + Weather + GitHub + Nextcloud + qBittorrent + RSS Feed + Speedtest Tracker + System Health Monitoring + System Resources + Services Map + Media Stack | 43 | diff --git a/wiki/concepts/_index.md b/wiki/concepts/_index.md index 92e326f..6cbf86c 100644 --- a/wiki/concepts/_index.md +++ b/wiki/concepts/_index.md @@ -216,5 +216,13 @@ | [[wiki/concepts/figma-fig-binary-format\|Figma .fig File — Proprietary Binary Format; Assets Extractable, Layout Data Not]] | `.fig` = ZIP with proprietary binary `canvas.fig` (magic bytes `fig-kiwij`); image assets extractable via `unzip`, but layer/component/text data requires Figma REST API or Desktop | daily/2026-05-10.md | 2026-05-10 | | [[wiki/concepts/vite-base-url-slash-concatenation]] | `${BASE_URL}file.png` works in dev (BASE_URL="/") but breaks in prod — always use explicit slash or `new URL(file, BASE_URL).href` | daily/2026-05-10.md | 2026-05-10 | | [[wiki/concepts/adguard-parallel-instances-diagnostic]] | Two AdGuard instances on different IPs — how to identify the real production one vs empty placeholder; fix is router-side DNS update | daily/2026-05-03.md | 2026-05-03 | +| [[wiki/concepts/payload-cms-overrideaccess-bypass]] | `overrideAccess: true` in Payload CMS completely bypasses access control — critical anti-pattern in webhook handlers; unauthenticated lead modification | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/nextjs-unstable-cache-force-dynamic]] | `unstable_cache` + tag revalidation vs `force-dynamic` — ISR pattern for shared rarely-changing data; 99%+ reduction in external API calls | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/map-ratelimiter-memory-leak]] | In-memory Map rate limiters grow unbounded — `setInterval` eviction with `.unref()` prevents memory leak and test runner hang | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/vitest-module-level-env-testing]] | `vi.stubEnv` + `vi.resetModules()` + dynamic `await import()` required when module reads env vars at load time | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/webhook-replay-attack-prevention]] | Timestamp ±5min validation + callId deduplication (DB or Redis NX) to prevent webhook replay attacks — HMAC alone is not sufficient | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/secrets-git-history-purge]] | `git-filter-repo --invert-paths --path .env` permanently removes secrets from history; force push required; all clones must be redone | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/payload-cms-push-dev-prod]] | `push: process.env.NODE_ENV === 'development'` — auto-schema in dev, migration-only in prod; `push: false` default silently breaks fresh DB | daily/2026-05-09.md | 2026-05-09 | +| [[wiki/concepts/cloudflare-proxied-domain-npm-subpath]] | Cloudflare-only domains have no NPM nginx vhost — Custom Locations fail; solution is a separate subdomain with its own NPM Proxy Host | daily/2026-05-09.md | 2026-05-09 | diff --git a/wiki/concepts/cloudflare-proxied-domain-npm-subpath.md b/wiki/concepts/cloudflare-proxied-domain-npm-subpath.md new file mode 100644 index 0000000..0e84bfc --- /dev/null +++ b/wiki/concepts/cloudflare-proxied-domain-npm-subpath.md @@ -0,0 +1,88 @@ +--- +name: cloudflare-proxied-domain-npm-subpath +description: Cloudflare-proxied domains without an NPM proxy host have no nginx config — NPM Custom Locations cannot be added; requires a separate subdomain +type: concept +--- + +# Nginx Proxy Manager — Custom Locations Don't Work for Cloudflare-Only Domains + +## The Setup + +A typical homelab has two categories of external domains: + +1. **NPM-managed domains**: Domain → Cloudflare → NPM → internal service. NPM generates a full nginx vhost config. +2. **Cloudflare-only domains**: Domain → Cloudflare → directly to internal IP (Cloudflare Full/Strict mode, or just DNS). No NPM entry exists. + +## The Problem + +NPM's "Custom Locations" feature lets you add path-based routing to an existing Proxy Host: + +``` +nextcloud.ai-impress.com/push/ → http://localhost:7867 +``` + +But this only works if a Proxy Host entry for `nextcloud.ai-impress.com` exists in NPM. If the domain goes directly through Cloudflare to the Nextcloud server, **NPM has no nginx config file for it**. There is no vhost to attach a Custom Location to. + +You cannot add a Custom Location for a domain that has no NPM Proxy Host. + +## Real Example — Nextcloud Signaling Server + +The `nextcloud.ai-impress.com` domain was accessed via Cloudflare without a matching NPM proxy host. Attempting to add `nextcloud.ai-impress.com/push/` as a custom location to route WebSocket traffic to the `strukturag/nextcloud-spreed-signaling` server failed — there was no NPM entry to attach it to. + +### Incorrect Approach + +``` +Existing NPM host: (none for nextcloud.ai-impress.com) +Attempt: Add Custom Location → /push/ → localhost:8083 +Result: Error — no proxy host to add location to +``` + +### Correct Approach + +Create a **separate subdomain** with its own NPM Proxy Host entry: + +``` +NPM Proxy Host #39: + Domain: signal.ai-impress.com + Forward: localhost:8083 (signaling server) + SSL: Let's Encrypt + WebSocket support: enabled + +Nextcloud config (config.php): + 'signaling_servers' => [ + ['url' => 'wss://signal.ai-impress.com'] + ] +``` + +## General Rule + +| Domain routing | NPM Custom Locations available? | +|---------------|----------------------------------| +| Domain → NPM → service (NPM manages SSL) | Yes | +| Domain → Cloudflare → NPM → service | Yes (NPM has the entry) | +| Domain → Cloudflare → direct IP (no NPM) | **No** — NPM has no vhost for this domain | + +## HTTP 404 from Signaling Server (Not an Error) + +A related diagnostic: the `strukturag/nextcloud-spreed-signaling` server returns HTTP 404 when accessed without a WebSocket upgrade header. This is normal and expected behavior — the server only handles WebSocket connections, not plain HTTP GET requests. A 404 from a curl to the signaling URL does not indicate misconfiguration. + +```bash +curl https://signal.ai-impress.com/spreed +# → 404 Not Found (expected — WebSocket upgrade required) +``` + +Test the WebSocket connection instead: +```bash +wscat -c wss://signal.ai-impress.com/spreed +``` + +## Container Config Path Mismatch (strukturag/nextcloud-spreed-signaling) + +The signaling server container expects its config at `/config/server.conf`, not `/etc/signaling/server.conf`. Mount accordingly: + +```yaml +volumes: + - ./signaling/server.conf:/config/server.conf:ro +``` + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/map-ratelimiter-memory-leak.md b/wiki/concepts/map-ratelimiter-memory-leak.md new file mode 100644 index 0000000..7433b4c --- /dev/null +++ b/wiki/concepts/map-ratelimiter-memory-leak.md @@ -0,0 +1,94 @@ +--- +name: map-ratelimiter-memory-leak +description: In-memory Map rate limiters grow unbounded in long-running Node.js servers — need setInterval eviction with .unref() +type: concept +--- + +# In-Memory Map Rate Limiter — Memory Leak and Eviction Pattern + +## The Pattern + +A token-bucket rate limiter using a JavaScript `Map` is a common, dependency-free implementation: + +```ts +const buckets = new Map() + +export function rateLimit(ip: string): boolean { + const now = Date.now() + let bucket = buckets.get(ip) + if (!bucket) { + bucket = { tokens: MAX_TOKENS, lastRefill: now } + buckets.set(ip, bucket) + } + // refill logic... + return bucket.tokens > 0 +} +``` + +## The Problem + +Every unique IP address creates a new `Map` entry. Entries are never deleted. In a long-running server this means: + +- Each unique client = permanent memory allocation +- After weeks of production traffic, thousands of stale entries occupy RAM +- In a high-traffic scenario (CDN scraping, bot traffic) this becomes a slow memory leak that eventually causes OOM + +The `Map` also holds strong references, preventing GC of the bucket objects. + +## The Fix — `setInterval` with `.unref()` + +```ts +const REFILL_INTERVAL_MS = 60_000 // 1 minute + +// Evict stale buckets that haven't been touched in 2 refill cycles +setInterval(() => { + const cutoff = Date.now() - REFILL_INTERVAL_MS * 2 + for (const [ip, bucket] of buckets) { + if (bucket.lastRefill < cutoff) { + buckets.delete(ip) + } + } +}, REFILL_INTERVAL_MS).unref() +// ^^^^^^^^ +// .unref() prevents the interval from keeping the process alive +// if this is the only remaining async operation (important for tests) +``` + +### Why `.unref()`? + +`setInterval` holds an event loop reference. If you don't call `.unref()`: +- Test runners (Vitest, Jest) will hang after tests complete +- `process.exit()` won't fire until the interval fires or is cleared +- In serverless/edge environments the process may not terminate cleanly + +`.unref()` makes the interval "passive" — it runs if the event loop is still active from other work, but it won't prevent the process from exiting. + +## Sizing the Eviction Window + +The cutoff of `REFILL_INTERVAL_MS * 2` gives each IP two full refill cycles of inactivity before eviction. A returning client after eviction simply gets a fresh full-token bucket (as if first visit), which is the intended behavior. + +For aggressive rate limiting (e.g., blocking brute-force) make the window longer to ensure blocked IPs stay tracked: + +```ts +const EVICTION_WINDOW_MS = 24 * 60 * 60 * 1000 // 24 hours +setInterval(() => { + const cutoff = Date.now() - EVICTION_WINDOW_MS + for (const [ip, bucket] of buckets) { + if (bucket.lastRefill < cutoff) buckets.delete(ip) + } +}, 60_000).unref() +``` + +## Production Recommendation + +For production rate limiting under real load, prefer Redis-backed solutions (`ioredis` + sliding window script, or `@upstash/ratelimit`) that: +- Survive server restarts +- Work across multiple instances +- Have built-in expiry via Redis TTL + +Use in-memory `Map` for: +- Development / single-instance deployments +- Low-traffic internal tooling +- When Redis is unavailable and you control the instance count + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/nextjs-unstable-cache-force-dynamic.md b/wiki/concepts/nextjs-unstable-cache-force-dynamic.md new file mode 100644 index 0000000..c9e79bb --- /dev/null +++ b/wiki/concepts/nextjs-unstable-cache-force-dynamic.md @@ -0,0 +1,86 @@ +--- +name: nextjs-unstable-cache-force-dynamic +description: unstable_cache with tag-based revalidation vs force-dynamic — ISR pattern for rarely-changing external API data in Next.js 15 +type: concept +--- + +# Next.js 15 — `unstable_cache` vs `force-dynamic` for External API Routes + +## The Problem with `force-dynamic` + +```ts +// BAD — disables ALL caching, hits DB + external API on every request +export const dynamic = 'force-dynamic' + +export async function GET() { + const tariffs = await fetchFromExternalAPI() + // ... +} +``` + +`export const dynamic = 'force-dynamic'` is a segment-level directive that opts the entire route out of Next.js's Data Cache. Every request triggers: +1. A fresh fetch to the external API +2. A fresh DB query +3. Full server rendering overhead + +This is appropriate for user-specific data that changes per-request (e.g., cart contents, auth status). It is **wrong** for data that is the same for all users and changes infrequently (e.g., pricing tariffs, product catalogs). + +## The Fix — `unstable_cache` with Tag-Based Revalidation + +```ts +import { unstable_cache } from 'next/cache' + +// Remove: export const dynamic = 'force-dynamic' + +const getCachedTariffs = unstable_cache( + async () => { + const [ezyTariffs, dbTariffs] = await Promise.all([ + fetchFromExternalAPI(), + payload.find({ collection: 'tariffs', limit: 1000 }), + ]) + return mergeTariffs(ezyTariffs, dbTariffs.docs) + }, + ['tariffs'], // cache key + { + revalidate: 300, // revalidate every 5 minutes (seconds) + tags: ['tariffs'], // tag for on-demand invalidation + } +) + +export async function GET() { + const tariffs = await getCachedTariffs() + return Response.json(tariffs) +} +``` + +### On-Demand Revalidation (e.g., after admin update) + +```ts +import { revalidateTag } from 'next/cache' + +// Call from a webhook or admin action after tariffs change +revalidateTag('tariffs') +``` + +## Choosing the Right Strategy + +| Data type | Strategy | Why | +|-----------|----------|-----| +| Per-user, changes every request | `force-dynamic` or `no-store` | No shared cache possible | +| Shared, changes rarely (tariffs, config) | `unstable_cache` with `revalidate` | Serve stale, refresh in background | +| Static, never changes at runtime | Default (static generation) | Build-time bake-in | +| Shared, changes after specific events | `unstable_cache` with `tags` | Invalidate precisely | + +## Naming Note + +`unstable_cache` is stable in practice despite the prefix — Vercel ships production apps on it. The `unstable_` prefix signals it may be renamed in a future API stabilization. The `cache()` function (React 19's `use cache` directive) is the eventual successor. + +## Performance Impact + +For a tariff endpoint called N times/minute: +- `force-dynamic`: N × (external API latency + DB query) +- `unstable_cache` (5 min TTL): 1 × (external API latency + DB query) per 5 min + N × cache read + +At 100 req/min this is a ~99.7% reduction in external API calls. + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/payload-cms-overrideaccess-bypass.md b/wiki/concepts/payload-cms-overrideaccess-bypass.md new file mode 100644 index 0000000..8d26058 --- /dev/null +++ b/wiki/concepts/payload-cms-overrideaccess-bypass.md @@ -0,0 +1,99 @@ +--- +name: payload-cms-overrideaccess-bypass +description: overrideAccess:true in Payload CMS completely bypasses the access control system — critical anti-pattern in webhook handlers +type: concept +--- + +# Payload CMS — `overrideAccess: true` Bypasses All Access Control + +## What It Does + +Passing `overrideAccess: true` to any Payload local API call (`payload.find()`, `payload.update()`, `payload.create()`, etc.) tells Payload to **skip the collection's `access` functions entirely**. No role check, no ownership check, no field-level restriction — the call succeeds as if it came from a superadmin. + +```ts +// This bypasses ALL access control — any caller can modify any lead +await payload.update({ + collection: 'leads', + id: leadId, + data: { status: 'converted' }, + overrideAccess: true, // ← NEVER do this in a public-facing handler +}) +``` + +## Why It Exists + +`overrideAccess` is intended for: +- Server-side seed scripts +- Internal background jobs that run with inherent trust +- Admin CLI utilities + +It is **not** intended for request handlers that accept untrusted input. + +## The Anti-Pattern + +Using `overrideAccess: true` inside a webhook route (or any publicly-reachable API route) means: + +1. **HMAC validation failure** (or intentional bypass) → attacker can manipulate Payload documents without any identity. +2. Even with HMAC in place — if HMAC is mis-implemented (e.g., empty-string secret fallback) the access control is your last defense. `overrideAccess: true` removes it. + +### Real Example (Shumiland project — `src/app/api/binotel/webhook/route.ts`) + +```ts +// BAD: overrideAccess:true in a webhook handler +const lead = await payload.find({ + collection: 'leads', + where: { phone: { equals: phone } }, + overrideAccess: true, // ← attacker can enumerate leads +}) + +await payload.update({ + collection: 'leads', + id: lead.docs[0].id, + data: { callStatus: newStatus }, + overrideAccess: true, // ← attacker can overwrite any lead +}) +``` + +## The Fix + +Remove `overrideAccess: true` and either: + +**Option A — use a system-level user identity (recommended for webhooks):** + +```ts +// Create a service account user and use req.user +const systemUser = await getSystemUser() // cached, read once +await payload.update({ + collection: 'leads', + id: lead.docs[0].id, + data: { callStatus: newStatus }, + user: systemUser, // proper identity context + // no overrideAccess +}) +``` + +**Option B — use `req.payload` with the Payload admin user if running inside a Next.js route handler:** + +```ts +// The req object from NextRequest has no Payload user attached +// You need to explicitly provide identity +``` + +**Minimum mitigation** if full fix isn't feasible: scope `overrideAccess: true` only to the narrowest possible read (by phone + exact match), and verify HMAC rigorously before reaching it. + +## Related Vulnerabilities + +When combined with other gaps this becomes critical: + +| Gap | Effect when combined with overrideAccess | +|-----|------------------------------------------| +| `BINOTEL_HMAC_SECRET ?? ''` empty fallback | HMAC skipped entirely → full unauthenticated write | +| No timestamp validation | Replayed requests can re-trigger status changes indefinitely | +| HMAC logged on failure | Secret leaks to any log aggregation system | + +## See Also + +- [[wiki/concepts/webhook-replay-attack-prevention]] — timestamp + nonce deduplication +- [[wiki/concepts/secrets-git-history-purge]] — if the HMAC secret was committed + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/payload-cms-push-dev-prod.md b/wiki/concepts/payload-cms-push-dev-prod.md new file mode 100644 index 0000000..717c646 --- /dev/null +++ b/wiki/concepts/payload-cms-push-dev-prod.md @@ -0,0 +1,100 @@ +--- +name: payload-cms-push-dev-prod +description: Payload CMS postgresAdapter push:false vs push:true — conditional schema auto-push for dev, migration-only for prod +type: concept +--- + +# Payload CMS — `push` Config for `postgresAdapter` (Dev vs Prod) + +## What `push` Does + +In `payload.config.ts`, `postgresAdapter` has a `push` option: + +```ts +import { postgresAdapter } from '@payloadcms/db-postgres' + +export default buildConfig({ + db: postgresAdapter({ + pool: { connectionString: process.env.DATABASE_URI }, + push: true, // or false + }), +}) +``` + +- **`push: true`** — Payload calls `drizzle-kit push` at startup, auto-syncing the schema to the DB. No migrations needed. Schema changes are applied immediately on every restart. +- **`push: false`** (default) — Payload uses `drizzle-kit migrate` style. Schema is only updated by running explicit migration files in `migrationDir`. + +## The Critical Gotcha + +**`push` defaults to `false`.** + +After wiping the database volume (e.g., during development or staging reset), `push: false` means **tables are never created**. The admin panel returns SQL errors: + +``` +relation "users" does not exist +relation "payload_preferences" does not exist +``` + +This is a silent failure — there are no startup errors, just 500s when the admin panel tries to query tables that don't exist. + +## The Correct Pattern + +```ts +postgresAdapter({ + pool: { connectionString: process.env.DATABASE_URI }, + // Auto-push in dev (no migration workflow needed) + // Migration-only in prod (controlled, auditable, reversible) + push: process.env.NODE_ENV === 'development', + migrationDir: './migrations', +}) +``` + +### Why Not Always `push: true`? + +In production `push: true` means every deployment auto-alters the database schema. This is dangerous because: +- Column renames/drops are immediate and irreversible on the running DB +- No migration history → can't rollback schema changes +- Concurrent deployments can race on schema changes + +### Why Not Always `push: false`? + +In development `push: false` requires running `npx payload migrate` after every collection change. This adds friction and is easy to forget, leading to confusing "table not found" errors after volume resets. + +## Running Migrations in Production + +```bash +# Generate a migration from current schema changes +npx payload migrate:create --name add-status-to-leads + +# Apply all pending migrations +npx payload migrate + +# Check migration status +npx payload migrate:status +``` + +## Docker Compose Entrypoint Pattern + +Run migrations before starting the server in production: + +```dockerfile +# Dockerfile +CMD ["sh", "-c", "npx payload migrate && node server.js"] +``` + +Or in `docker-compose.yml`: + +```yaml +services: + web: + command: sh -c "npx payload migrate && node .next/standalone/server.js" + depends_on: + db: + condition: service_healthy +``` + +## Related Scaffold Issue + +When generating a new Payload + Next.js project, the scaffold (as of Payload 3.x) may omit `push: false` in the production config template. Always check `payload.config.ts` before deploying to production. + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/secrets-git-history-purge.md b/wiki/concepts/secrets-git-history-purge.md new file mode 100644 index 0000000..77ba450 --- /dev/null +++ b/wiki/concepts/secrets-git-history-purge.md @@ -0,0 +1,115 @@ +--- +name: secrets-git-history-purge +description: git-filter-repo to permanently remove committed secrets from git history — secrets are recoverable even in private repos until purged +type: concept +--- + +# Secrets Committed to Git History — Detection and Permanent Purge + +## Why Git History Is Dangerous + +`git log`, `git show`, `git blame` all expose history. A secret committed in commit `abc123` and later deleted in commit `def456` is still fully visible via `git show abc123:path/to/.env`. + +In private repos this matters because: +- Anyone with repo read access (current or past collaborators, CI tokens) can retrieve it +- GitHub/Bitbucket may have indexed it +- Any clone or fork carries the full history + +**Rotation is mandatory immediately; purge removes the attack surface permanently.** + +## Detection — Finding the Commits + +```bash +# Find commits that introduced .env or secrets files +git log --all --full-history -- .env +git log --all --full-history -- "*.env*" + +# Show the actual content of a specific commit's .env +git show abc123:.env + +# Search all commits for a specific string (slow on large repos) +git log -p --all | grep -i "POSTGRES_PASSWORD\|SECRET\|API_KEY" +``` + +## Purge — `git-filter-repo` + +`git-filter-repo` is the modern replacement for `git filter-branch` (the old approach is slower, more error-prone, and deprecated by Git maintainers). + +```bash +# Install +pip install git-filter-repo +# or: brew install git-filter-repo + +# Remove specific files from ALL history +git filter-repo --invert-paths --path .env +git filter-repo --invert-paths --path .env.local +git filter-repo --invert-paths --path config/secrets.yml + +# Remove multiple paths in one pass (more efficient) +git filter-repo \ + --invert-paths \ + --path .env \ + --path .env.local \ + --path docker-compose.override.yml +``` + +`--invert-paths` means "remove these paths" (keep everything else). + +## After the Purge + +```bash +# Verify the file is gone from history +git log --all --full-history -- .env +# Should return nothing + +# Force push ALL branches (required — history was rewritten) +git push --force --all +git push --force --tags + +# Notify all collaborators: their local clones are now diverged +# They MUST delete and re-clone — they cannot merge/rebase from old history +``` + +## Mandatory Steps After Any Secret Exposure + +1. **Rotate immediately** — change the secret value before purging history +2. **Purge** — remove from git history +3. **Force push** — push rewritten history to remote +4. **Re-clone** — all team members must delete their local clone and re-clone +5. **Audit access logs** — check if the leaked secret was used during the exposure window +6. **Add to `.gitignore`** — prevent re-committing + +## Real Example (Shumiland project — commits eaa200f and ae9a7cf) + +``` +eaa200f — committed .env with POSTGRES_PASSWORD, PAYLOAD_SECRET +ae9a7cf — committed docker-compose.yml with BINOTEL_HMAC_SECRET +``` + +Even though `.env` was added to `.gitignore` in a later commit, the secrets remain visible in those two commits until `git-filter-repo` is run. + +## Prevention + +```gitignore +# .gitignore +.env +.env.* +!.env.example +docker-compose.override.yml +``` + +Pre-commit hook (add to `.pre-commit-config.yaml`): +```yaml +- repo: https://github.com/Yelp/detect-secrets + rev: v1.4.0 + hooks: + - id: detect-secrets +``` + +Or use `git-secrets` from AWS: +```bash +git secrets --install +git secrets --register-aws +``` + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/vitest-module-level-env-testing.md b/wiki/concepts/vitest-module-level-env-testing.md new file mode 100644 index 0000000..5973cdf --- /dev/null +++ b/wiki/concepts/vitest-module-level-env-testing.md @@ -0,0 +1,99 @@ +--- +name: vitest-module-level-env-testing +description: vi.stubEnv + vi.resetModules() + dynamic await import() required when testing modules that read env vars at load time +type: concept +--- + +# Vitest — Testing Modules That Read Env Vars at Import Time + +## The Problem + +Many Next.js route files or Node.js modules read `process.env` variables at module load time: + +```ts +// src/app/api/binotel/webhook/route.ts +const SECRET = process.env.BINOTEL_HMAC_SECRET ?? '' +// ^^^ +// Read ONCE when the module is imported — not on each request +``` + +A standard Vitest test that sets `process.env.BINOTEL_HMAC_SECRET = ''` after importing the module **does not affect** the `SECRET` constant — it was already captured. + +```ts +// BAD — env var set after module is already cached +import { POST } from '../route' + +test('rejects empty secret', async () => { + process.env.BINOTEL_HMAC_SECRET = '' // too late — module already imported + const res = await POST(makeRequest()) + expect(res.status).toBe(500) // might not behave as expected +}) +``` + +## The Fix — `vi.stubEnv` + `vi.resetModules()` + Dynamic Import + +```ts +import { vi, describe, beforeEach, afterEach, it, expect } from 'vitest' + +describe('webhook route — empty secret', () => { + beforeEach(() => { + vi.stubEnv('BINOTEL_HMAC_SECRET', '') // 1. Set env var before import + vi.resetModules() // 2. Clear module cache + }) + + afterEach(() => { + vi.unstubAllEnvs() // 3. Restore env vars after test + vi.resetModules() // 4. Clean slate for next test + }) + + it('returns 500 when secret is empty', async () => { + // 5. Dynamic import AFTER stubEnv + resetModules + const { POST } = await import('../route') + const res = await POST(makeRequest()) + expect(res.status).toBe(500) + }) +}) +``` + +### Why This Works + +| Step | Effect | +|------|--------| +| `vi.stubEnv('KEY', 'value')` | Sets `process.env.KEY` and records it for later restoration | +| `vi.resetModules()` | Clears Vitest's module registry — next `import()` re-executes the module | +| `await import('../route')` | Module re-executes with new `process.env` values, captures updated constants | +| `vi.unstubAllEnvs()` | Restores `process.env` to original values | + +## Testing `timingSafeEqual` Length Mismatch + +A related gotcha: `crypto.timingSafeEqual(a, b)` throws a `RangeError` if buffers have different lengths. Most implementations add a length-mismatch fast-path: + +```ts +function safeCompare(a: string, b: string): boolean { + if (a.length !== b.length) return false // ← must be tested explicitly + return crypto.timingSafeEqual(Buffer.from(a), Buffer.from(b)) +} +``` + +Test the fast-path separately: + +```ts +it('returns false for different-length strings without throwing', () => { + expect(safeCompare('abc', 'abcdef')).toBe(false) + // Would throw RangeError if fast-path is missing +}) +``` + +## ESM Module Testing + +If the module under test is pure ESM (has `"type": "module"` and uses `import`/`export` without CommonJS), it cannot be `require()`-d in a test. The dynamic `await import()` pattern works because Vitest runs in ESM mode itself. + +If a module uses top-level `await` at import time (e.g., connects to DB), `resetModules()` + re-import will re-run that connection on every test — use `vi.mock` to stub the dependency instead. + +## Pattern Summary + +``` +stubEnv → resetModules → await import() → test → unstubAllEnvs → resetModules +``` + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/concepts/webhook-replay-attack-prevention.md b/wiki/concepts/webhook-replay-attack-prevention.md new file mode 100644 index 0000000..5ce8279 --- /dev/null +++ b/wiki/concepts/webhook-replay-attack-prevention.md @@ -0,0 +1,107 @@ +--- +name: webhook-replay-attack-prevention +description: Timestamp validation (±5min window) + request deduplication (per-phone nonce/ID) to prevent replay attacks on webhook endpoints +type: concept +--- + +# Webhook Security — Replay Attack Prevention + +## What Is a Replay Attack? + +An attacker captures a legitimate, HMAC-signed webhook request and re-sends it later. Even if HMAC validation passes (the signature is valid), the request should be rejected because it was already processed. + +## Two-Layer Defense + +### Layer 1 — Timestamp Validation (±5 Minute Window) + +The webhook sender includes a timestamp in the payload or headers. The receiver rejects requests where the timestamp is more than 5 minutes old: + +```ts +const MAX_AGE_MS = 5 * 60 * 1000 // 5 minutes + +export async function POST(req: NextRequest) { + const body = await req.json() + const { timestamp, callId, phone, ...data } = body + + // 1. Timestamp validation + const age = Date.now() - new Date(timestamp).getTime() + if (Math.abs(age) > MAX_AGE_MS) { + return Response.json({ error: 'Request expired' }, { status: 400 }) + } + + // 2. HMAC validation (must happen before any DB access) + const signature = req.headers.get('X-Binotel-Signature') ?? '' + if (!verifyHmac(body, signature)) { + return Response.json({ error: 'Invalid signature' }, { status: 401 }) + } + + // 3. Deduplication (see Layer 2) +} +``` + +### Layer 2 — Request Deduplication (Nonce / Idempotency Key) + +Even within the 5-minute window, an attacker could replay a request several times. Use a nonce or call ID stored in the DB: + +```ts +// After timestamp + HMAC validation... + +// Check if this exact call was already processed +const existing = await payload.find({ + collection: 'processed-webhooks', + where: { callId: { equals: callId } }, + limit: 1, +}) + +if (existing.docs.length > 0) { + return Response.json({ status: 'duplicate' }, { status: 200 }) + // Return 200 (not 4xx) to prevent the sender retrying +} + +// Process the webhook... + +// Record the nonce AFTER successful processing +await payload.create({ + collection: 'processed-webhooks', + data: { callId, processedAt: new Date().toISOString() }, +}) +``` + +### Alternative — Per-Phone Nonce with TTL + +If a separate `processed-webhooks` collection is too heavy, use a Redis SET with TTL: + +```ts +const nonce = `webhook:${callId}` +const alreadySeen = await redis.set(nonce, '1', 'EX', 300, 'NX') +// NX = only set if not exists, returns null if already set +if (alreadySeen === null) { + return Response.json({ status: 'duplicate' }, { status: 200 }) +} +``` + +## Vulnerability Without These Measures + +| Missing defense | Attack vector | +|----------------|---------------| +| No timestamp check | Attacker stores a signed request, replays it days later | +| No deduplication | Attacker replays a fresh request N times in the valid window | +| Both missing | Any captured request can be replayed indefinitely | + +## HMAC Is Necessary But Not Sufficient + +HMAC proves the request came from someone with the secret. It does NOT prove the request is being received for the first time. Replay prevention is always layered on top of HMAC, not a replacement. + +## Standard Timestamp Header Pattern (Stripe-style) + +Many providers (Stripe, GitHub, Binotel) include the timestamp in: +- A dedicated header: `Stripe-Signature: t=1234567890,v1=abc...` +- The JSON payload body + +The HMAC is computed over `timestamp + "." + raw_body` so the timestamp is signed along with the payload. This prevents an attacker from modifying the timestamp on a captured request. + +## See Also + +- [[wiki/concepts/payload-cms-overrideaccess-bypass]] — must fix access control to prevent exploits even if replay prevention is bypassed + +**Source:** daily/2026-05-09.md | 2026-05-09 diff --git a/wiki/log.md b/wiki/log.md index 1c4c6d2..c46764b 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -1,6 +1,21 @@ # Build Log +## [2026-05-10T12:00:00+02:00] compile | daily/2026-05-09.md +- Source: daily/2026-05-09.md +- Articles created (8): + - [[wiki/concepts/payload-cms-overrideaccess-bypass]] — `overrideAccess: true` bypasses all Payload access control; critical anti-pattern in webhook handlers + - [[wiki/concepts/nextjs-unstable-cache-force-dynamic]] — `unstable_cache` + tag revalidation vs `force-dynamic`; ISR for shared rarely-changing data + - [[wiki/concepts/map-ratelimiter-memory-leak]] — in-memory Map rate limiters grow unbounded; `setInterval` eviction with `.unref()` + - [[wiki/concepts/vitest-module-level-env-testing]] — `vi.stubEnv` + `vi.resetModules()` + dynamic `await import()` for env-at-load-time modules + - [[wiki/concepts/webhook-replay-attack-prevention]] — timestamp ±5min validation + callId deduplication to prevent replay attacks + - [[wiki/concepts/secrets-git-history-purge]] — `git-filter-repo --invert-paths --path .env` permanent secret removal; force push + re-clone required + - [[wiki/concepts/payload-cms-push-dev-prod]] — conditional `push: NODE_ENV === 'development'`; `push: false` default silently breaks fresh DB + - [[wiki/concepts/cloudflare-proxied-domain-npm-subpath]] — Cloudflare-only domains have no NPM vhost; Custom Locations fail; separate subdomain required +- Articles updated: none +- Index updates: [[wiki/concepts/_index]] (166→174, +8 entries); [[wiki/_master-index]] (concepts 166→174) +- Session coverage: Nextcloud signaling server config path mismatch, NPM Custom Location limitation for Cloudflare-proxied domains, Payload CMS security audit (overrideAccess, secrets in git, HMAC logging, replay attacks), Payload CMS performance (force-dynamic, N+1 queries, O(N×M) map lookup, rate limiter leak), Vitest env-at-load-time testing patterns, docker-compose credential mismatch, payload push:false gotcha + ## [2026-05-10T23:00:00+02:00] compile | daily/2026-05-03.md (addendum — Glance widgets + dual AdGuard) - Source: daily/2026-05-03.md - Articles created: [[wiki/concepts/adguard-parallel-instances-diagnostic]]