diff --git a/.obsidian/graph.json b/.obsidian/graph.json index 1bd25ab..1246aa7 100644 --- a/.obsidian/graph.json +++ b/.obsidian/graph.json @@ -17,6 +17,6 @@ "repelStrength": 10, "linkStrength": 1, "linkDistance": 250, - "scale": 0.45971825916974024, + "scale": 0.7108694604315626, "close": true } \ No newline at end of file diff --git a/99 Daily/2026-04-15.md b/99 Daily/2026-04-15.md index 022674a..c70faef 100644 --- a/99 Daily/2026-04-15.md +++ b/99 Daily/2026-04-15.md @@ -14,3 +14,15 @@ tags: [daily] - 10:27 — Claude Code session ended | `memory-compiler` - 10:32 — Claude Code session ended | `memory-compiler` - 10:33 — Claude Code session ended | `memory-compiler` +- 10:40 (1min) — Claude Code session ended | `aimpress` +- 10:41 — Claude Code session ended | `aimpress` +- 10:42 — Claude Code session ended | `aimpress` +- 10:46 | `aimpress` + - **Asked:** Remove Ollama and all associated files and data. + - **Done:** Uninstalled Ollama, dependencies, models (6.1 GB), configs, logs, and background services. +- 10:46 | `aimpress` + - **Asked:** Set up an Obsidian logging system that automatically captures all Claude Code interactions across projects with proper session documentation. + - **Done:** Identified that the Stop hook needs to be rewritten to parse transcripts and call Haiku for structured entries, rather than just logging timestamps. +- 10:47 | `aimpress` + - **Asked:** Set up a system to automatically log all Claude Code interactions across projects into Obsidian with proper structure and formatting. + - **Done:** Configured structured logging format that captures "Asked" and "Done" for each session, tested it on current transcript, and verified it works with Haiku-powered processing (~$0.001 per session). diff --git a/wiki/_master-index.md b/wiki/_master-index.md index fa076f9..91e667c 100644 --- a/wiki/_master-index.md +++ b/wiki/_master-index.md @@ -19,6 +19,10 @@ This 3-hop pattern works for hundreds of articles without vector search. | Topic | Description | Articles | |-------|-------------|----------| | [[wiki/obsidian-rag/_index\|obsidian-rag/]] | Karpathy's LLM wiki method — Obsidian RAG, setup, vs true RAG | 3 | +| [[wiki/projects-overview/_index\|projects-overview/]] | All 36 Oliver Agency projects — navigable table with stack + server | 1 | +| [[wiki/tech-patterns/_index\|tech-patterns/]] | Recurring tech stacks: FastAPI, React/Vite, Next.js, Azure AD, AI, Box, One2Edit | 9 | +| [[wiki/architecture/_index\|architecture/]] | Cross-cutting architectural patterns: Docker Compose, multi-agent AI, GCP timeout, RAG, hotfolder | 5 | +| [[wiki/client-knowledge/_index\|client-knowledge/]] | Per-client notes for Ford, H&M, L'Oréal (2+ projects each) | 3 | | [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 0 | | [[wiki/connections/_index\|connections/]] | Cross-cutting insights linking 2+ concepts | 0 | | [[wiki/qa/_index\|qa/]] | Filed answers to queries (saved with `--file-back`) | 0 | diff --git a/wiki/architecture/_index.md b/wiki/architecture/_index.md new file mode 100644 index 0000000..f3067f4 --- /dev/null +++ b/wiki/architecture/_index.md @@ -0,0 +1,29 @@ +--- +title: "Architecture Patterns Index" +description: "Cross-cutting architectural decisions across Oliver Agency projects" +tags: [index, architecture] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Architecture Patterns + +Cross-cutting architectural decisions that appear in multiple Oliver projects. + +## Patterns + +| Pattern | Description | Projects | +|---------|-------------|---------| +| [[wiki/architecture/docker-compose-fullstack\|docker-compose-fullstack]] | Standard multi-service Docker Compose deployment | ~20 projects | +| [[wiki/architecture/multi-agent-ai-systems\|multi-agent-ai-systems]] | Parallel specialist agents + lead synthesizer | Mod Comms, Semblance, Enterprise Nexus | +| [[wiki/architecture/gcp-deployment-lb-timeout\|gcp-deployment-lb-timeout]] | GCP 30s LB timeout — WebSocket → HTTP polling fix | Mod Comms, Semblance | +| [[wiki/architecture/rag-architecture\|rag-architecture]] | RAG: Firecrawl → AI structuring → Qdrant → LLM synthesis | Enterprise Nexus, Sandbox NotebookLM | +| [[wiki/architecture/hotfolder-daemon\|hotfolder-daemon]] | Box folder monitoring daemon with systemd | Ford QC, Ford SFTP | + +## Key Architectural Decisions + +1. **Docker Compose** — default deployment for all multi-service projects +2. **HTTP polling over WebSocket** — mandatory on GCP (30s LB timeout) +3. **AI pre-structuring before RAG indexing** — improves retrieval quality +4. **Hotfolder + archive pattern** — prevents reprocessing in Box automations +5. **DEV_AUTH_BYPASS** — skip Azure AD in local dev, always use real auth in production diff --git a/wiki/architecture/docker-compose-fullstack.md b/wiki/architecture/docker-compose-fullstack.md new file mode 100644 index 0000000..289f65d --- /dev/null +++ b/wiki/architecture/docker-compose-fullstack.md @@ -0,0 +1,128 @@ +--- +title: "Docker Compose Fullstack Deployment" +aliases: [docker-compose, docker, deployment, devops] +tags: [docker, docker-compose, deployment, fullstack] +sources: [01 Projects/gmal-scope-builder, 01 Projects/modcomms, 01 Projects/enterprise-ai-hub-nexus, 01 Projects/video-accessibility, 01 Projects/sandbox-notebookllamalm-nextjs] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Docker Compose Fullstack Deployment + +The standard Oliver deployment model. Every service (frontend, backend, DB, Redis, workers) runs as a Docker Compose service. + +## Key Takeaways +- `docker compose up --build` is the standard single-command deploy for ~20 projects +- Pin port to non-default for PostgreSQL (5433, not 5432) to avoid local conflicts +- Backend and frontend build separately — rebuild only what changed to save time +- Always copy `.env.example` → `.env` before first run +- `deploy.sh` script in some projects (GMAL) automates env setup + build + +## When to Use +Any project with 2+ services (e.g., API + DB, or frontend + backend + DB). + +## Standard Compose Patterns + +### 3-Service Stack (Most Common) +```yaml +# docker-compose.yml +services: + frontend: + build: ./frontend + ports: ["3000:3000"] + depends_on: [backend] + + backend: + build: ./backend + ports: ["8000:8000"] + env_file: .env + depends_on: [db] + + db: + image: postgres:16 + ports: ["5433:5432"] # non-default external port + volumes: ["pgdata:/var/lib/postgresql/data"] + environment: + POSTGRES_DB: mydb + POSTGRES_USER: myuser + POSTGRES_PASSWORD: mypass + +volumes: + pgdata: +``` + +### Full AI Stack (Enterprise Nexus) +```yaml +services: + frontend: # Next.js 14 + backend: # FastAPI + Celery + postgres: # PostgreSQL 16 (nexus-postgres) + redis: # Redis 7 (nexus-redis) + qdrant: # Qdrant vector DB (nexus-qdrant) + worker: # Celery worker + beat: # Celery beat scheduler +``` + +### With nginx Proxy (DeckForge) +```yaml +services: + nginx: # :80 → routes to frontend or backend + frontend: # Next.js :3000 (internal) + backend: # FastAPI :8000 (internal) +``` + +## Dev Commands +```bash +# Standard workflow +cp .env.example .env # First time only +docker compose up --build # Build + start all + +# Faster iteration +docker compose build backend && docker compose up -d backend +docker compose logs backend --tail=50 + +# DB operations +docker compose exec db psql -U myuser -d mydb +docker compose exec backend alembic upgrade head + +# Backup DB +docker compose exec db pg_dump -U myuser -d mydb > backups/dump.sql +``` + +## Production Deploy (Sandbox NotebookLM Pattern) +```bash +# On server (optical-web-1) +git pull origin main +docker compose build backend # or frontend or both +docker compose up -d +docker compose logs backend --tail=50 +``` + +## Port Conventions +| Service | Internal Port | External Port | +|---------|--------------|---------------| +| FastAPI backend | 8000 | 8000 or 8001 | +| React/Vite frontend | 3000 | 3000 or 3010 | +| Next.js | 3000 | 3000 | +| PostgreSQL | 5432 | **5433** (avoid local conflict) | +| Redis | 6379 | 6379 | +| Qdrant | 6333 | 6333 | + +## Projects Using This Pattern +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — 7-service compose (frontend, backend, postgres, redis, qdrant, worker, beat) +- [[01 Projects/gmal-scope-builder/GMAL Scope Builder|GMAL Scope Builder]] — 3-service (frontend :3010, backend :8001, db :5433) +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — 3-service, deployed to GCP +- [[01 Projects/video-accessibility/Video Accessibility Platform|Video Accessibility]] — multi-worker compose with Celery + MongoDB +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — Next.js 15 + FastAPI + PostgreSQL 18 + Redis 7 + +## Gotchas & Lessons +- Don't use port 5432 externally for PostgreSQL — conflicts with local Postgres installs +- `.env.example` → `.env` step is almost always required; document it in README +- `docker compose up --build` every time is safe but slow — use targeted `build backend` for iteration +- Celery workers need their own container with separate `command:` override +- Volume names should be prefixed with project name to avoid sharing data between projects + +## Related +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — the backend being deployed +- [[wiki/tech-patterns/redis-celery-worker-queue|redis-celery-worker-queue]] — worker services +- [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]] — GCP-specific constraints diff --git a/wiki/architecture/gcp-deployment-lb-timeout.md b/wiki/architecture/gcp-deployment-lb-timeout.md new file mode 100644 index 0000000..4a38f42 --- /dev/null +++ b/wiki/architecture/gcp-deployment-lb-timeout.md @@ -0,0 +1,83 @@ +--- +title: "GCP Deployment & 30s Load Balancer Timeout" +aliases: [gcp, gcp-timeout, load-balancer-timeout, websocket-polling] +tags: [gcp, deployment, websocket, polling, timeout, architecture] +sources: [01 Projects/modcomms, 01 Projects/semblance] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# GCP Deployment & 30s Load Balancer Timeout + +GCP's HTTP(S) Load Balancer drops connections after 30 seconds. Long-running AI tasks that relied on WebSocket streaming were breaking in production. + +## Key Takeaways +- GCP HTTP(S) LB has a **hard 30-second timeout** on backend connections +- WebSocket connections through GCP LB are killed at 30s → client sees disconnect +- Fix: replace WebSocket delivery with **HTTP polling** (client polls `/status` endpoint) +- This affects both Mod Comms and Semblance — same root cause, same fix +- Task results must be stored server-side (DB or Redis) so the client can retrieve them on poll + +## The Problem +``` +Client → WebSocket → GCP LB → FastAPI/Quart backend + [30 seconds pass] + GCP LB KILLS CONNECTION ← AI task not done yet +Client sees error, no result delivered +``` + +## The Fix: HTTP Polling +``` +1. Client POSTs task → backend returns { task_id } +2. Client polls GET /tasks/{task_id}/status every 2s +3. Backend stores result in DB/Redis when task completes +4. On 'complete' status, client fetches full result +``` + +```python +# FastAPI endpoint pattern +@router.post("/analyze") +async def start_analysis(proof: UploadFile): + task_id = str(uuid4()) + background_tasks.add_task(run_ai_analysis, task_id, proof) + return {"task_id": task_id} + +@router.get("/tasks/{task_id}/status") +async def get_status(task_id: str): + result = await db.get_task(task_id) + return {"status": result.status, "data": result.data if result.done else None} +``` + +```js +// Frontend polling +async function pollTask(taskId) { + const { status, data } = await api.get(`/tasks/${taskId}/status`) + if (status === 'complete') return data + await sleep(2000) + return pollTask(taskId) +} +``` + +## Projects Affected +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — switched WebSocket → REST polling (2026-03-18, critical fix) +- [[01 Projects/semblance/Semblance|Semblance]] — migrated all async LLM routes (2026-03-23, critical fix) + +## Server Details +- Both Mod Comms and Semblance deploy to GCP +- Sandbox NotebookLM and Cinema Studio Pro are on `ai-sandbox.oliver.solutions` (optical-web-1) — different infrastructure, different rules + +## Secondary GCP Notes +- WebSocket keepalive heartbeat should be ≤25s (Mod Comms tuned to 10s) +- Consider Cloud Run for serverless deployments — has configurable timeout up to 3600s +- GCP LB timeout is configurable in Backend Service settings, but changing it requires infra access + +## Gotchas & Lessons +- Increasing WebSocket heartbeat (25s → 10s) was not enough — had to drop WebSocket entirely +- Don't assume WebSocket works through GCP LB without testing with real LB (not just local) +- Any future GCP project with AI tasks >5s should default to HTTP polling architecture +- Socket.IO fallback (polling mode) is NOT the same as HTTP polling for results — Socket.IO still goes through LB + +## Related +- [[wiki/architecture/multi-agent-ai-systems|multi-agent-ai-systems]] — the pattern that triggers long tasks +- [[wiki/tech-patterns/redis-celery-worker-queue|redis-celery-worker-queue]] — storing results for polling +- [[wiki/tech-patterns/python-ai-agents|python-ai-agents]] — the long-running AI calls diff --git a/wiki/architecture/hotfolder-daemon.md b/wiki/architecture/hotfolder-daemon.md new file mode 100644 index 0000000..a036066 --- /dev/null +++ b/wiki/architecture/hotfolder-daemon.md @@ -0,0 +1,97 @@ +--- +title: "Hotfolder Daemon — Box Monitoring" +aliases: [hotfolder, daemon, box-monitor, systemd-daemon] +tags: [hotfolder, daemon, systemd, box, ford, automation, python] +sources: [01 Projects/ford_qc, 01 Projects/ford-gechub-sftp] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Hotfolder Daemon — Box Monitoring + +Persistent background daemon that watches a Box folder for new files, processes them automatically, and archives the originals. + +## Key Takeaways +- Ford QC runs as a **systemd service** on a production server — survives reboots +- Hotfolder pattern: poll Box folder → detect new files → process → upload result → archive original +- Archive processed files immediately to prevent reprocessing +- CLI mode (single file) and daemon mode (continuous) — same core logic, different entry points +- Ford SFTP uses daemon mode (`python main.py --daemon`) without systemd (simpler) + +## When to Use +Any automation that must respond to files dropped in a Box folder without human intervention. + +## Architecture + +### Ford QC Hotfolder +``` +Box folder 332861865120 (hotfolder, poll every N seconds) + ↓ detect new ZIP +ford_qc_box_hotfolder_process.py + ↓ +qc_engine.py → 13 QC check modules (checks/) + ↓ +JSON results → HTML report (html_reporter.py) + ↓ +Upload HTML report → Box 332864939558 (reports) +Archive ZIP → Box 332861653811 (archive) +``` + +### Ford SFTP Daemon +``` +Box folders (PROD / EDU / QA — 3 separate targets) + ↓ poll for new ZIPs +ford_qc_box_hotfolder_process.py + ↓ +Download ZIP → Upload to GECHUB SFTP (paramiko) + ↓ +Archive in Box + ↓ +Email notification (Mailgun) +``` + +## Systemd Service (Ford QC) +```bash +# /etc/systemd/system/ford-qc-hotfolder.service +[Unit] +Description=Ford QC Hotfolder Daemon + +[Service] +ExecStart=/usr/bin/python3 /path/to/ford_qc_box_hotfolder_process.py +Restart=always + +[Install] +WantedBy=multi-user.target + +# Commands +sudo systemctl start ford-qc-hotfolder.service +sudo systemctl enable ford-qc-hotfolder.service # start on boot +sudo systemctl status ford-qc-hotfolder.service +``` + +## Projects Using This Pattern +- [[01 Projects/ford_qc/Ford QC System|Ford QC System]] — systemd service, 13 QC check modules, Box JWT auth, HTML reports +- [[01 Projects/ford-gechub-sftp/Ford SFTP Transfer|Ford SFTP Transfer]] — `--daemon` flag, 3 environments, SFTP upload, Mailgun notifications + +## QC Module Pattern (Ford QC) +```python +# qc_module.py +class QCModule: + def check(self, zip_path: str, profile: dict) -> dict: + """Returns {"passed": bool, "issues": [...]}""" + +# 13 modules in checks/: +# image resolution, format, file size, layer depth, colour existence, +# linking validation, MEC/BAU compliance, powertrain validation, lifestyle inventory +``` + +## Gotchas & Lessons +- Always archive processed files — the daemon will reprocess on next poll if not archived +- Box JWT config (`ford_box_config.json`) must not be committed to git +- Ranger ptvl pattern is configurable via profile JSON — don't hardcode (fix: 2026-03-16) +- Three environments (PROD/EDU/QA) have completely separate Box folder IDs and SFTP targets +- Mailgun integration: keep email templates simple — Mailgun rejects complex HTML occasionally + +## Related +- [[wiki/tech-patterns/box-api-integration|box-api-integration]] — Box auth and folder IDs +- [[wiki/client-knowledge/ford|ford]] — Ford-specific context and environment details diff --git a/wiki/architecture/multi-agent-ai-systems.md b/wiki/architecture/multi-agent-ai-systems.md new file mode 100644 index 0000000..f2d2716 --- /dev/null +++ b/wiki/architecture/multi-agent-ai-systems.md @@ -0,0 +1,86 @@ +--- +title: "Multi-Agent AI Systems" +aliases: [multi-agent, ai-agents, parallel-agents] +tags: [ai, multi-agent, architecture, gemini, gpt] +sources: [01 Projects/modcomms, 01 Projects/semblance, 01 Projects/enterprise-ai-hub-nexus] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Multi-Agent AI Systems + +Pattern where multiple specialized AI agents run in parallel, with a lead/orchestrator agent synthesizing results. + +## Key Takeaways +- Parallel specialist agents + lead synthesizer = reliable, multi-perspective analysis +- HTTP polling (not WebSocket) for delivering async agent results on GCP +- Each agent should have a focused, single-concern prompt (Legal, Brand, Tone, Channel) +- Autonomous orchestration needs explicit "next speaker" logic to prevent infinite loops +- Background task execution (`ai_runner_service.py`) keeps agents non-blocking + +## When to Use +- Content review requiring multiple compliance dimensions (legal + brand + tone) +- Focus group simulation (multi-persona conversations) +- Any task benefiting from multiple independent perspectives before synthesis + +## Architecture Patterns + +### Pattern 1: Parallel Specialists + Lead (Mod Comms) +``` +Input (proof image/PDF) + ↓ +┌──────────────────────────────────┐ +│ Agent 1: Legal compliance │ +│ Agent 2: Brand adherence │ ← run in PARALLEL +│ Agent 3: Tone of voice │ +│ Agent 4: Channel suitability │ +└──────────────────────────────────┘ + ↓ +Lead Agent: synthesize verdict + ↓ +Result (via HTTP polling) +``` + +### Pattern 2: Autonomous Multi-Persona (Semblance) +``` +Input (discussion brief) + ↓ +Persona generator (Gemini) → N personas + ↓ +Conversation controller + ├── conversation_decision_service.py ← next speaker logic + ├── conversation_context_service.py ← shared state + history + └── ai_runner_service.py ← background task execution + ↓ +Socket.IO → frontend (real-time) + ↓ +Theme extraction + analytics +``` + +### Pattern 3: RAG + Structured AI (Enterprise Nexus) +``` +Document upload → Firecrawl crawl + ↓ +AI content structuring (pre-indexing) + ↓ +Qdrant vector DB (10-page batch merge) + ↓ +Query → vector search → LLM synthesis +``` + +## Projects Using This Pattern +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — 4 parallel specialist agents + lead synthesizer (Gemini Pro/Flash) +- [[01 Projects/semblance/Semblance|Semblance]] — Autonomous focus group with N personas (Gemini 3 Pro, GPT-4.1, GPT-5.2) +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — RAG with structured AI pre-processing (Qdrant) + +## Gotchas & Lessons +- GCP 30s LB timeout kills streaming delivery — always use HTTP polling for agent results (see [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]]) +- Semblance: naive vs aware datetime crash — always use timezone-aware datetimes in async contexts +- "Next speaker" logic in autonomous mode must have termination conditions to prevent infinite loops +- Cross-loop WebSocket emit in Semblance was unreliable — polling fallback was more stable +- Orphaned vectors in Qdrant need periodic cleanup (Enterprise Nexus has a "Purge orphaned vectors" button) + +## Related +- [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]] — critical deployment constraint +- [[wiki/architecture/rag-architecture|rag-architecture]] — retrieval-augmented generation +- [[wiki/tech-patterns/python-ai-agents|python-ai-agents]] — model selection + structured output diff --git a/wiki/architecture/rag-architecture.md b/wiki/architecture/rag-architecture.md new file mode 100644 index 0000000..a362c0a --- /dev/null +++ b/wiki/architecture/rag-architecture.md @@ -0,0 +1,100 @@ +--- +title: "RAG Architecture (Retrieval-Augmented Generation)" +aliases: [rag, vector-search, knowledge-retrieval, qdrant] +tags: [rag, qdrant, vector-search, llamaindex, firecrawl, architecture] +sources: [01 Projects/enterprise-ai-hub-nexus, 01 Projects/sandbox-notebookllamalm-nextjs, 01 Projects/Oliver-ai-bot_2.0] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# RAG Architecture + +Knowledge retrieval pipeline: ingest documents → chunk + embed → store in vector DB → retrieve relevant chunks → LLM synthesis. + +## Key Takeaways +- Qdrant is the vector DB of choice for Oliver RAG projects +- AI content structuring **before** indexing improves retrieval quality (Enterprise Nexus) +- Firecrawl `/v1/crawl` handles recursive website crawling for knowledge ingestion +- Batch processing (10 pages at a time) for document merging avoids context overflow +- Orphaned vectors accumulate over time — need periodic cleanup +- LlamaIndex abstracts multi-model RAG (Sandbox NotebookLM) + +## When to Use +When users need to query a large body of documents in natural language. + +## Architecture + +### Enterprise Nexus RAG Pipeline +``` +Sources: + - File upload (PDF, Word, etc.) + - SharePoint sync (Celery beat) + - Website crawl (Firecrawl /v1/crawl) + ↓ +AI content structuring (pre-processing pass) + ↓ +Qdrant (vector store) + ↓ on query: +Vector similarity search → top-k chunks + ↓ +LLM synthesis (prompt + chunks) + ↓ +Response to user +``` + +### Sandbox NotebookLM RAG +``` +Document upload → LlamaIndex ingestion + ↓ +llm_factory.get_llm_by_type() → multi-model + ↓ +7 studio generators (flashcards, quiz, mindmap, slides, report, infographic, datatable) + ↓ +ElevenLabs (podcast audio synthesis) +``` + +## Key Components + +### Qdrant Operations +```python +# Upsert vectors +client.upsert(collection_name="knowledge", points=[ + PointStruct(id=doc_id, vector=embedding, payload={"text": chunk, "source": filename}) +]) + +# Search +hits = client.search(collection_name="knowledge", query_vector=query_embedding, limit=5) +chunks = [hit.payload["text"] for hit in hits] + +# Cleanup orphaned vectors +client.delete(collection_name="knowledge", points_selector=orphaned_ids) +``` + +### Firecrawl Site Crawl +```python +response = firecrawl.crawl_url(url, params={"crawlOptions": {"maxDepth": 3}}) +# Returns list of {url, markdown, metadata} +``` + +### AI Pre-structuring (Enterprise Nexus) +- Run LLM pass on raw document before chunking +- Extracts structured sections, improves chunk boundaries +- Batch: 10 pages at a time to stay within context window + +## Projects Using This Pattern +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise AI Hub Nexus]] — Qdrant + Firecrawl + AI pre-structuring + SharePoint sync + Celery +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — LlamaIndex multi-model + 7 studio generators + ElevenLabs podcast +- [[01 Projects/Oliver-ai-bot_2.0/Oliver AI Bot 2.0|Oliver AI Bot 2.0]] — RAG mode (85% complete) + +## Gotchas & Lessons +- Qdrant vectors are NOT automatically deleted when source documents are removed — implement orphaned vector cleanup +- AI pre-structuring significantly improves RAG quality but adds ~2-3s per document at ingest +- SharePoint sync token refresh must be proactive — Celery beat jobs run after tokens may have expired +- M365 delegated scopes + `offline_access` required for refresh tokens (Enterprise Nexus) +- `max_tokens` 8192 was too low for from-template PPTX generation in Sandbox — increased to 16000 + +## Related +- [[wiki/tech-patterns/redis-celery-worker-queue|redis-celery-worker-queue]] — SharePoint sync scheduling +- [[wiki/tech-patterns/azure-ad-msal-auth|azure-ad-msal-auth]] — M365 auth for SharePoint +- [[wiki/tech-patterns/python-ai-agents|python-ai-agents]] — LLM integration details +- [[wiki/architecture/multi-agent-ai-systems|multi-agent-ai-systems]] — related AI pattern diff --git a/wiki/client-knowledge/_index.md b/wiki/client-knowledge/_index.md new file mode 100644 index 0000000..871101f --- /dev/null +++ b/wiki/client-knowledge/_index.md @@ -0,0 +1,30 @@ +--- +title: "Client Knowledge Index" +description: "What Vadym knows about working with specific clients — tech preferences, quirks, and constraints" +tags: [index, client-knowledge] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Client Knowledge + +Per-client notes for clients with 2+ active projects. Covers tech preferences, constraints, and lessons learned. + +## Clients + +| Client | Projects | Key Tech | Article | +|--------|---------|---------|---------| +| Ford | Ford QC, Ford SFTP | Box API, SFTP, systemd, Python | [[wiki/client-knowledge/ford\|ford]] | +| H&M | O2E Tool, EMS Report | One2Edit API, Python, JSON | [[wiki/client-knowledge/hm\|hm]] | +| L'Oréal | Global Kickoff, SLA Calculator | Box API, PHP, Make.com, Docker | [[wiki/client-knowledge/loreal\|loreal]] | + +## Single-Project Clients +These clients have only one project — context lives in the project note: +- **3M** → [[01 Projects/3m-portal/3M OMG Portal|3M OMG Portal]] — One2Edit proxy +- **Barclays** → [[01 Projects/modcomms/Mod Comms|Mod Comms]] — AI proof review, GCP +- **BAIC** → [[01 Projects/baic_dashboard/BAIC Dashboard|BAIC Dashboard]] — Make.com + Azure AD +- **PIMCO** → [[01 Projects/pimco-charts/PIMCO Chart Generator|PIMCO Charts]] — SVG chart generation +- **Ferrero** → [[01 Projects/ferrero-ac-creator/Ferrero AC Booking|Ferrero AC Booking]] — Box API + CSV +- **Solventum** → [[01 Projects/solventum-image-metadata/Solventum Image Metadata|Solventum Metadata]] — OpenAI + enterprise +- **LUSA** → [[01 Projects/lusa-back-planner/LUSA Back Planner|LUSA Back Planner]] — Timeline + PDF +- **Dow Jones / WSJ** → [[01 Projects/wsj-filenaming/WSJ File Naming Tool|WSJ File Naming]] — Gemini AI naming diff --git a/wiki/client-knowledge/ford.md b/wiki/client-knowledge/ford.md new file mode 100644 index 0000000..247e5b7 --- /dev/null +++ b/wiki/client-knowledge/ford.md @@ -0,0 +1,66 @@ +--- +title: "Ford — Client Knowledge" +aliases: [ford, ford-motor] +tags: [client, ford, box, sftp, qc, automation] +sources: [01 Projects/ford_qc, 01 Projects/ford-gechub-sftp] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Ford + +Oliver Agency works with Ford on asset pack automation — QC validation and delivery to Ford's GECHUB SFTP system. + +## Key Takeaways +- Ford has **3 environments**: PROD, EDU, QA — each has separate Box folders and SFTP targets +- Asset packs are ZIP files — QC runs 13 checks before delivery +- Ford uses GECHUB as their SFTP-based delivery system +- Box JWT service account (`ford_box_config.json`) — never commit this file +- The Ford QC daemon runs as a **systemd service** on the production server + +## Projects +| Project | Purpose | Entry Point | +|---------|---------|-------------| +| [[01 Projects/ford_qc/Ford QC System\|Ford QC System]] | QC validation of asset pack ZIPs | systemd or `python qc_engine.py` | +| [[01 Projects/ford-gechub-sftp/Ford SFTP Transfer\|Ford SFTP Transfer]] | Box → GECHUB SFTP transfer | `python main.py [--daemon]` | + +## Environment Details +| Env | Purpose | Box Folder | SFTP Target | +|-----|---------|-----------|-------------| +| PROD | Production assets | Box/PROD | GECHUB PROD | +| EDU | Educational assets | Box/EDU | GECHUB EDU | +| QA | QA testing | Box/QA | GECHUB QA | + +## Box Folder IDs (Ford QC) +| Folder | ID | +|--------|-----| +| Input hotfolder | 332861865120 | +| Reports output | 332864939558 | +| Archive | 332861653811 | + +## Ford QC Checks (13 modules) +- Image resolution, format, file size +- Layer depth, colour existence +- Linking validation +- MEC/BAU compliance +- Powertrain validation +- Lifestyle inventory +- HTML report generation + +## Tech Preferences +- Box API (JWT auth) — file exchange platform +- SFTP (GECHUB) — delivery system +- systemd for production daemons +- Email notifications via Mailgun +- QC profiles in JSON (configurable per asset type) +- HTML reports (not PDFs) + +## Quirks & Lessons +- Ranger ptvl pattern must be configurable via QC profile — don't hardcode it (fix: 2026-03-16) +- Always archive processed ZIPs immediately — daemon reprocesses if not archived +- Ford has 3 completely separate environments — changes in PROD profiles don't affect EDU/QA +- GECHUB SFTP credentials differ per environment + +## Related +- [[wiki/architecture/hotfolder-daemon|hotfolder-daemon]] — the daemon pattern +- [[wiki/tech-patterns/box-api-integration|box-api-integration]] — Box details diff --git a/wiki/client-knowledge/hm.md b/wiki/client-knowledge/hm.md new file mode 100644 index 0000000..b0f30c4 --- /dev/null +++ b/wiki/client-knowledge/hm.md @@ -0,0 +1,53 @@ +--- +title: "H&M — Client Knowledge" +aliases: [hm, h&m, handm] +tags: [client, hm, one2edit, ems, product-review] +sources: [01 Projects/hm-o2e-tool, 01 Projects/hm_ems_report] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# H&M + +Oliver Agency works with H&M on document translation/relinking (One2Edit platform) and product data review workflows. + +## Key Takeaways +- H&M uses **One2Edit** as their document management/translation platform +- EMS product review is multi-language (up to 4 languages simultaneously) +- Both H&M tools are lightweight — no Docker, minimal infrastructure +- Product data lives in a master JSON structure; edits save back to master + +## Projects +| Project | Purpose | Entry Point | +|---------|---------|-------------| +| [[01 Projects/hm-o2e-tool/HM O2E Tool\|H&M O2E Tool]] | Image relinking + document export via One2Edit | Static / `python -m http.server` | +| [[01 Projects/hm_ems_report/HM EMS Report Tool\|H&M EMS Report Tool]] | Multi-language product review and approval | `python server.py` | + +## H&M O2E Tool Capabilities +- Automated image relinking (by File or by Folder) +- Document export from One2Edit +- Image status checking +- Static — no backend required + +## H&M EMS Review Tool Capabilities +- Side-by-side review of product names + prices in up to 4 languages +- Inline editing +- Approval workflow → saves to `Master_Json/` +- Full change log +- Campaign images with enlarge preview + +## Tech Preferences +- One2Edit API (`oliver.one2edit.com/v3/Api.php`) +- Lightweight tools — no Docker, no complex build +- JSON as data store (not a database) +- Python simple HTTP server for local serving + +## Quirks & Lessons +- One2Edit is shared with 3M — same platform, different use cases +- H&M O2E tool can run fully static (open `index.html`) for most operations +- EMS tool: master JSON is the source of truth — backup before any bulk operations +- Multi-language display: columns for each language, all visible simultaneously + +## Related +- [[wiki/tech-patterns/one2edit-api|one2edit-api]] — One2Edit platform details +- [[wiki/client-knowledge/ford|ford]] — another client using Box-based workflows diff --git a/wiki/client-knowledge/loreal.md b/wiki/client-knowledge/loreal.md new file mode 100644 index 0000000..1213304 --- /dev/null +++ b/wiki/client-knowledge/loreal.md @@ -0,0 +1,56 @@ +--- +title: "L'Oréal — Client Knowledge" +aliases: [loreal, l'oreal, loreal-global] +tags: [client, loreal, box, make, php, sla, global-to-local] +sources: [01 Projects/loreal-global-kickoff, 01 Projects/loreal-sla-calculator] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# L'Oréal + +Oliver Agency works with L'Oréal on campaign asset management (global → local transformation) and SLA timeline tooling. + +## Key Takeaways +- L'Oréal uses **Box** for campaign asset exchange and folder hierarchy encodes campaign numbers +- **Make.com** is used as the automation backbone (webhook on submission) +- The "Global to Local" workflow transforms global campaign CSVs into regional market CSVs +- SLA calculator replaces Excel — 9 brief types, 8 stages, Gantt view +- PHP backend for the global kickoff tool (not Python/Node) + +## Projects +| Project | Purpose | Entry Point | +|---------|---------|-------------| +| [[01 Projects/loreal-global-kickoff/Loreal Global Kickoff\|L'Oréal OMG Assistant Global]] | Asset submission + global-to-local CSV transform | PHP server | +| [[01 Projects/loreal-sla-calculator/Loreal SLA Calculator\|L'Oréal SLA Calculator]] | SLA timeline calculator for eCom Content Factory PMs | `docker compose up` | + +## L'Oréal OMG Assistant Capabilities +- **Master Global Asset Submission** — submit Box assets with metadata +- Auto-populate campaign number from Box folder hierarchy +- **Global to Local** — transform global CSVs into regional market CSVs +- Webhook to Make.com on submission + +## L'Oréal SLA Calculator Capabilities +- Replaces Excel for SLA timeline management +- 9 brief types × 8 stages +- Gantt chart view + date verdict +- CSV + iCal export +- Client Estimator (`client.html`) — simplified 4-input version + +## Tech Preferences +- PHP for server-rendered web apps (not Python or Node) +- Box API for file storage and folder-based campaign organization +- Make.com for automation/workflow orchestration +- CSV as data exchange format for global → local transforms +- Docker for SLA calculator (simpler tool) + +## Quirks & Lessons +- Campaign number is encoded in the Box folder hierarchy — must traverse parent folders to extract it +- Make.com webhook triggers on every submission — design idempotent handlers +- The "Global to Local" transform needs to understand regional market codes in CSV headers +- SLA calculator has two UX modes: full PM calculator vs simplified client estimator +- L'Oréal uses eCom Content Factory workflow — their PM team are the primary users of the SLA tool + +## Related +- [[wiki/tech-patterns/box-api-integration|box-api-integration]] — Box API patterns +- [[wiki/client-knowledge/ford|ford]] — another Box-heavy client diff --git a/wiki/projects-overview/_index.md b/wiki/projects-overview/_index.md new file mode 100644 index 0000000..6b70feb --- /dev/null +++ b/wiki/projects-overview/_index.md @@ -0,0 +1,79 @@ +--- +title: "Projects Overview" +description: "All Oliver Agency projects — navigable map with tech stack and status" +tags: [projects, oliver, index] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Projects Overview + +All active projects at `/Volumes/SSD/Projects/Oliver/`. One row per project. + +## Key Takeaways +- 36 projects total; majority use Docker Compose as deploy mechanism +- FastAPI + React/Vite + Docker is the dominant full-stack pattern +- Azure AD / MSAL appears in 6+ projects (Oliver internal tools) +- GCP is the primary production server for external-facing tools +- Box API used by Ford, Ferrero, L'Oréal integrations + +## Project Map + +| Project | Client | Status | Tech Stack | Server / Run | +|---------|--------|--------|-----------|--------------| +| [[01 Projects/3m-portal/3M OMG Portal\|3M OMG Portal]] | 3M | Active | Node.js, Vanilla JS, One2Edit API | `node server.js` :3000 | +| [[01 Projects/apac-ops-bot/APAC Ops Bot\|APAC Ops Bot]] | Oliver | Active | TBD | TBD | +| [[01 Projects/DevOps_Click_UP_sync/DevOps ClickUp Sync\|DevOps ClickUp Sync]] | Oliver | Active | Python, FastAPI, SQLite, Tailwind, Docker | `docker compose up` :8080 | +| [[01 Projects/Oliver-ai-bot_2.0/Oliver AI Bot 2.0\|Oliver AI Bot 2.0]] | Oliver | Active | Next.js, FastAPI, Python, Docker | `docker compose up` | +| [[01 Projects/ac-helper/AC Helper\|AC Helper]] | Oliver | Active | Python, Gemini, HTML/JS, Tailwind, Docker | `docker compose up` | +| [[01 Projects/ac-tool/AC Tool\|AC Tool]] | Oliver | Active | Docker | `docker compose up` | +| [[01 Projects/baic_dashboard/BAIC Dashboard\|BAIC Dashboard]] | BAIC | Active | React, Vite, Flask, Recharts, Azure AD, Make.com | `python app.py` + `npm run dev` | +| [[01 Projects/build-a-squad/Build A Squad\|Build A Squad]] | Oliver | Active | React 19, TypeScript, Vite, Gemini | `npm run dev` | +| [[01 Projects/cc-dashboard/CC Dashboard\|CC Dashboard]] | Oliver | Active | Docker | `docker compose up` | +| [[01 Projects/cinema-studio-pro/Cinema Studio Pro\|Cinema Studio Pro (Lux Studio)]] | Oliver | Active | React, Vite, PHP, Azure AD, Imagen 3, Veo, Kling | `./setup.sh` / FileZilla+SSH | +| [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus\|Enterprise AI Hub Nexus]] | Oliver | **Production** | Next.js 14, FastAPI, PostgreSQL, Redis, Qdrant, Azure AD, Celery, Firecrawl | `docker compose up` | +| [[01 Projects/ferrero-ac-creator/Ferrero AC Booking Tool\|Ferrero AC Booking Tool]] | Ferrero | Active | Node.js, HTML/JS, Box API | `node server.js` :3456 | +| [[01 Projects/ford-gechub-sftp/Ford SFTP Transfer\|Ford Asset Pack SFTP Transfer]] | Ford | Active | Python, Box API, SFTP, Mailgun | `python main.py [--daemon]` | +| [[01 Projects/ford_qc/Ford QC System\|Ford QC System]] | Ford | Active | Python, Box API, systemd | `systemctl start ford-qc-hotfolder` | +| [[01 Projects/gmal-scope-builder/GMAL Scope Builder\|GMAL Scope Builder]] | Oliver | Active | React, FastAPI, PostgreSQL, Claude Opus 4.6, Azure SSO, Docker | `docker compose up` :3010/:8001 | +| [[01 Projects/hm-o2e-tool/HM O2E Tool\|H&M O2E Tool]] | H&M | Active | HTML/JS, One2Edit API | Static / `python -m http.server` | +| [[01 Projects/hm_ems_report/HM EMS Report Tool\|H&M EMS Report Tool]] | H&M | Active | Python, HTML/JS, JSON | `python server.py` | +| [[01 Projects/homepage/Homepage Dashboard\|Homepage Dashboard]] | Oliver | Active | Node.js, Docker, YAML | `docker compose up` | +| [[01 Projects/loreal-global-kickoff/Loreal Global Kickoff\|L'Oréal OMG Assistant Global]] | L'Oréal | Active | PHP, Box API, Make.com | PHP server | +| [[01 Projects/loreal-sla-calculator/Loreal SLA Calculator\|L'Oréal SLA Calculator]] | L'Oréal | Active | HTML/JS, Docker, Node.js | `docker compose up` | +| [[01 Projects/lusa-back-planner/LUSA Back Planner\|LUSA Back Planner]] | LUSA | Active | React 19, TypeScript, Vite, Tailwind, shadcn/ui, pdfjs, jsPDF | `npm run dev` | +| [[01 Projects/md-to-word/MD to Word\|MD to Word Converter]] | Oliver | Active | Python | `python convert.py` | +| [[01 Projects/modcomms/Mod Comms\|Mod Comms]] | Barclays | Active | React, FastAPI, PostgreSQL, Gemini, Azure AD, Docker | `docker compose up` — GCP | +| [[01 Projects/olivas/OliVAS\|OliVAS]] | Oliver | Active | Python, FastAPI, Claude API, DeepGaze, Docker | `docker compose up` | +| [[01 Projects/oliver-ai-assistant/Oliver AI Assistant\|Oliver AI Assistant]] | Oliver | Active | TBD | TBD | +| [[01 Projects/pdf-accessibility/PDF Accessibility Checker\|PDF Accessibility Checker]] | Oliver | Active | Python, PHP, PostgreSQL, Redis, Claude, GCV, Docker | `docker compose up` | +| [[01 Projects/pimco-charts/PIMCO Chart Generator\|PIMCO Chart Generator]] | PIMCO | Active | Python, FastAPI, Claude API, SVG, Docker | `docker compose up` | +| [[01 Projects/ppt-tool/Oliver DeckForge\|Oliver DeckForge]] | Oliver | Active | Next.js, FastAPI, nginx, Docker | `docker compose up` | +| [[01 Projects/presenton/Presenton\|Presenton]] | Oliver | Active | Node.js, Docker | `docker compose up` | +| [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM\|Sandbox NotebookLM]] | Oliver | Active | Next.js 15, FastAPI, PostgreSQL, Redis, ElevenLabs, Docker | `docker compose up` — optical-web-1 | +| [[01 Projects/semblance/Semblance\|Semblance]] | Oliver | Active | React, Quart, Socket.IO, MongoDB, Gemini, OpenAI, Docker | `docker compose up` — GCP | +| [[01 Projects/smartcrop26/SmartCrop26\|SmartCrop26]] | Oliver | Active | React, TypeScript, Vite, Lovable | `npm run dev` | +| [[01 Projects/solventum-image-metadata/Solventum Image Metadata\|Solventum Image Metadata Tool]] | Solventum | Active | Python, FastAPI, OpenAI, Docker | `docker compose up` | +| [[01 Projects/video-accessibility/Video Accessibility Platform\|Video Accessibility Platform]] | Oliver | Active | React, FastAPI, Celery, MongoDB, Redis, Gemini, GCS, Docker | `docker compose up` | +| [[01 Projects/wsj-filenaming/WSJ File Naming Tool\|WSJ File Naming Tool]] | Dow Jones | Active | Python, HTML/JS, Gemini | `python server.py` | + +## Clients +| Client | Projects | Notes | +|--------|---------|-------| +| Oliver Internal | ~25 | Majority of projects | +| Ford | Ford QC, Ford SFTP | Box API + SFTP automation | +| H&M | O2E Tool, EMS Report | One2Edit + product review | +| L'Oréal | Global Kickoff, SLA Calculator | Box API + PHP | +| Barclays | Mod Comms | GCP deployment | +| 3M | OMG Portal | One2Edit proxy | +| BAIC | Dashboard | Make.com + Azure AD | +| PIMCO | Chart Generator | SVG publication charts | +| Ferrero | AC Booking | Box API + CSV | +| Solventum | Image Metadata | OpenAI + enterprise | +| LUSA | Back Planner | Timeline/PDF tool | +| Dow Jones / WSJ | File Naming Tool | Gemini AI naming | + +## Related +- [[wiki/tech-patterns/_index|tech-patterns/]] — recurring stack patterns +- [[wiki/architecture/_index|architecture/]] — cross-cutting architectural decisions +- [[wiki/client-knowledge/_index|client-knowledge/]] — per-client notes diff --git a/wiki/tech-patterns/_index.md b/wiki/tech-patterns/_index.md new file mode 100644 index 0000000..4be6b7a --- /dev/null +++ b/wiki/tech-patterns/_index.md @@ -0,0 +1,36 @@ +--- +title: "Tech Patterns Index" +description: "Recurring technology stacks across Oliver Agency projects" +tags: [index, tech-patterns] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Tech Patterns + +Recurring technology stacks used across Oliver Agency projects. Each article covers a specific pattern with projects, gotchas, and code examples. + +## Patterns + +| Pattern | Description | Projects | +|---------|-------------|---------| +| [[wiki/tech-patterns/fastapi-python-docker\|fastapi-python-docker]] | FastAPI + Python + Docker — dominant backend stack | GMAL, Mod Comms, Video Accessibility, OliVAS, PIMCO, Solventum, Enterprise Nexus | +| [[wiki/tech-patterns/react-vite-typescript\|react-vite-typescript]] | React + Vite + TypeScript — standard Oliver SPA frontend | GMAL, Mod Comms, Semblance, Video Accessibility, LUSA, Build-A-Squad, BAIC | +| [[wiki/tech-patterns/nextjs-fastapi-fullstack\|nextjs-fastapi-fullstack]] | Next.js + FastAPI — for complex multi-service AI platforms | Enterprise Nexus, Sandbox NotebookLM, Oliver AI Bot 2.0, DeckForge | +| [[wiki/tech-patterns/azure-ad-msal-auth\|azure-ad-msal-auth]] | Azure AD / MSAL SSO — Oliver internal auth standard | GMAL, Enterprise Nexus, Cinema Studio Pro, Mod Comms, BAIC, Sandbox | +| [[wiki/tech-patterns/python-ai-agents\|python-ai-agents]] | Claude / Gemini / OpenAI / LlamaIndex integration patterns | GMAL, Mod Comms, Semblance, OliVAS, PIMCO, Video Accessibility, Enterprise Nexus | +| [[wiki/tech-patterns/redis-celery-worker-queue\|redis-celery-worker-queue]] | Redis + Celery for async long-running tasks | Enterprise Nexus, Video Accessibility, PDF Accessibility | +| [[wiki/tech-patterns/box-api-integration\|box-api-integration]] | Box API for client asset workflows | Ford QC, Ford SFTP, L'Oréal, Ferrero | +| [[wiki/tech-patterns/one2edit-api\|one2edit-api]] | One2Edit translation platform API | 3M Portal, H&M O2E Tool | +| [[wiki/tech-patterns/nodejs-vanilla-proxy\|nodejs-vanilla-proxy]] | Node.js + Vanilla JS lightweight proxy tools | 3M Portal, Ferrero, Homepage | + +## Quick Decision Guide + +``` +New project → what stack? +├── Complex AI platform, multi-user → nextjs-fastapi-fullstack +├── Standard tool with UI → fastapi-python-docker + react-vite-typescript +├── Simple client portal / proxy → nodejs-vanilla-proxy +├── Static page, no backend → plain HTML/JS +└── Needs auth? → azure-ad-msal-auth +``` diff --git a/wiki/tech-patterns/azure-ad-msal-auth.md b/wiki/tech-patterns/azure-ad-msal-auth.md new file mode 100644 index 0000000..bfad756 --- /dev/null +++ b/wiki/tech-patterns/azure-ad-msal-auth.md @@ -0,0 +1,83 @@ +--- +title: "Azure AD / MSAL Authentication" +aliases: [azure-ad, msal, sso, pkce] +tags: [azure-ad, msal, auth, sso, pkce, security] +sources: [01 Projects/gmal-scope-builder, 01 Projects/enterprise-ai-hub-nexus, 01 Projects/cinema-studio-pro, 01 Projects/modcomms, 01 Projects/baic_dashboard, 01 Projects/sandbox-notebookllamalm-nextjs] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Azure AD / MSAL Authentication + +Oliver Agency standard SSO. MSAL handles token acquisition; backend validates via Azure AD JWT. + +## Key Takeaways +- All Oliver-internal tools requiring login use Azure AD (MSAL) +- MSAL v5 introduced breaking changes — `clearCache()` for logout, NOT `logout()` +- Send **ID token** to backend, NOT the Graph access token +- Use `DEV_AUTH_BYPASS=true` env var for local dev to skip auth +- Backend validates via `JWTValidator.php` (PHP) or Azure AD middleware (Python) +- PKCE flow is used for SPAs (Enterprise Nexus); standard OAuth2 for server-side + +## When to Use +Any Oliver internal tool that needs to know who the user is. Skip for pure tools/utilities. + +## Key Details + +### Frontend Config +```js +// authConfig.js / authConfig.ts +{ + auth: { + clientId: process.env.AZURE_CLIENT_ID, + authority: `https://login.microsoftonline.com/${TENANT_ID}`, + redirectUri: window.location.origin + }, + cache: { cacheLocation: "sessionStorage" } +} +``` + +### Required Env Vars +```env +AZURE_TENANT_ID= +AZURE_CLIENT_ID= +AZURE_CLIENT_SECRET= # server-side only +DEV_AUTH_BYPASS=true # local dev only +``` + +### Scopes +- **M365 integration** requires delegated scopes + `offline_access` (Enterprise Nexus) +- **Graph API** access requires explicit scope grants in Azure App Registration + +### MSAL v5 Breaking Changes +```js +// Old (MSAL v4): +instance.logout() +// New (MSAL v5): +instance.clearCache() +``` + +### Token Validation (Backend) +- **PHP:** `JWTValidator.php` + `AuthMiddleware.php` (Cinema Studio Pro) +- **Python:** Azure AD middleware checks Bearer token against tenant + +## Projects Using This Pattern +- [[01 Projects/gmal-scope-builder/GMAL Scope Builder|GMAL Scope Builder]] — MSAL v5, DEV_AUTH_BYPASS, Azure SSO +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — PKCE, delegated scopes, offline_access, M365 +- [[01 Projects/cinema-studio-pro/Cinema Studio Pro|Cinema Studio Pro]] — MSAL + JWTValidator.php backend +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — Azure AD, DISABLE_AUTH for local dev +- [[01 Projects/baic_dashboard/BAIC Dashboard|BAIC Dashboard]] — MSAL, Azure App Registration required +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — Microsoft SSO + signup/login + +## Gotchas & Lessons +- MSAL v5 `clearCache()` not `logout()` — fix was needed in GMAL (2026-03-30) +- Send ID token to backend, not Graph access token — GMAL bug fix (2026-03-30) +- `needs_reconsent` flag: if Graph token refresh fails, prompt user to re-login (Enterprise Nexus) +- Azure App Registration must have the delegated scopes explicitly added for M365 features +- `offline_access` scope required for refresh token to work (Enterprise Nexus M365) +- Proactive token refresh needed for long-running Celery jobs (Enterprise Nexus SharePoint sync) + +## Related +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — backend receiving tokens +- [[wiki/tech-patterns/react-vite-typescript|react-vite-typescript]] — frontend MSAL integration +- [[wiki/tech-patterns/nextjs-fastapi-fullstack|nextjs-fastapi-fullstack]] — Next.js PKCE flow diff --git a/wiki/tech-patterns/box-api-integration.md b/wiki/tech-patterns/box-api-integration.md new file mode 100644 index 0000000..55261c5 --- /dev/null +++ b/wiki/tech-patterns/box-api-integration.md @@ -0,0 +1,73 @@ +--- +title: "Box API Integration" +aliases: [box-api, box] +tags: [box, api, integration, ford, loreal, ferrero] +sources: [01 Projects/ford_qc, 01 Projects/ford-gechub-sftp, 01 Projects/loreal-global-kickoff, 01 Projects/ferrero-ac-creator] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Box API Integration + +Box is used as the file storage and workflow layer for several client integrations. Oliver Agency uses Box for asset handoff with Ford, L'Oréal, and Ferrero. + +## Key Takeaways +- Ford uses Box JWT auth (`ford_box_config.json`) — service account model +- L'Oréal and Ferrero use Box API for folder-based asset submission workflows +- Ford QC monitors specific Box folder IDs (hardcoded in profile JSON, not config) +- Box folder IDs are stable identifiers — use them, don't rely on folder names + +## When to Use +Any client project where files are exchanged via Box. Box is the primary asset handoff platform for Ford, L'Oréal, Ferrero. + +## Key Details + +### Auth Methods +| Project | Auth Method | Config | +|---------|------------|--------| +| Ford QC | JWT (service account) | `ford_box_config.json` | +| Ford SFTP | Box API | env vars | +| L'Oréal | Box API | env vars | +| Ferrero | Box API | env vars | + +### Ford Box Folder IDs +| Purpose | Folder ID | +|---------|-----------| +| Input hotfolder | 332861865120 | +| Reports output | 332864939558 | +| Archive (processed) | 332861653811 | + +### Common Operations +```python +# Download file from Box +client = Client(auth=JWTAuth.from_settings_file('ford_box_config.json')) +file_content = client.file(file_id).content() + +# Upload to Box +folder = client.folder(folder_id) +folder.upload_stream(file_stream, 'filename.zip') + +# List folder contents +items = client.folder(folder_id).get_items() +``` + +### Make.com Webhook Integration +L'Oréal Global Kickoff sends Box submission metadata to Make.com via webhook on form submit. + +## Projects Using This Pattern +- [[01 Projects/ford_qc/Ford QC System|Ford QC System]] — JWT auth, monitors 3 folder IDs, uploads HTML reports, archives processed files +- [[01 Projects/ford-gechub-sftp/Ford SFTP Transfer|Ford SFTP Transfer]] — Download from Box (3 envs), upload to GECHUB SFTP +- [[01 Projects/loreal-global-kickoff/Loreal Global Kickoff|L'Oréal Global Kickoff]] — Asset submission form + Make.com webhook +- [[01 Projects/ferrero-ac-creator/Ferrero AC Booking Tool|Ferrero AC Booking]] — "Send to OMG" saves CSV to Box folder + +## Gotchas & Lessons +- JWT config file (`ford_box_config.json`) must not be committed — it's a service account credential +- Box folder IDs are numeric strings — store in config, not hardcoded in logic +- Ford has 3 environments (PROD/EDU/QA) each with separate Box folders and SFTP targets +- Hotfolder daemon should archive processed files to prevent reprocessing (Ford QC pattern) +- Box webhooks have retry logic — design idempotent handlers + +## Related +- [[wiki/architecture/hotfolder-daemon|hotfolder-daemon]] — Ford QC Box monitoring pattern +- [[wiki/client-knowledge/ford|ford]] — Ford-specific context +- [[wiki/client-knowledge/loreal|loreal]] — L'Oréal-specific context diff --git a/wiki/tech-patterns/fastapi-python-docker.md b/wiki/tech-patterns/fastapi-python-docker.md new file mode 100644 index 0000000..878c704 --- /dev/null +++ b/wiki/tech-patterns/fastapi-python-docker.md @@ -0,0 +1,73 @@ +--- +title: "FastAPI + Python + Docker Stack" +aliases: [fastapi-docker, python-api] +tags: [fastapi, python, docker, backend] +sources: [01 Projects/gmal-scope-builder, 01 Projects/modcomms, 01 Projects/video-accessibility, 01 Projects/pdf-accessibility, 01 Projects/olivas, 01 Projects/pimco-charts, 01 Projects/solventum-image-metadata] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# FastAPI + Python + Docker Stack + +The dominant backend pattern at Oliver — Python FastAPI served via Docker Compose, typically paired with React/Vite frontend. + +## Key Takeaways +- Default backend choice for all new Oliver AI tools +- Uvicorn is the ASGI server; standard port is 8000 or 8001 +- Always use `asyncpg` + SQLAlchemy async when touching PostgreSQL +- Docker Compose is the standard deploy — backend + frontend + DB in one `compose.yml` +- `alembic` for database migrations (PostgreSQL projects) +- `uv` emerging as the package manager of choice (Sandbox NotebookLM uses Python 3.13 + uv) + +## When to Use +Any new Oliver internal tool needing a REST API backend, especially when AI integration is required. + +## Key Details +- **ASGI server:** Uvicorn (`uvicorn app.main:app --reload --host 0.0.0.0 --port 8000`) +- **Async DB:** SQLAlchemy async + asyncpg (PostgreSQL) or PyMongo (MongoDB) +- **Migrations:** Alembic (30+ revisions in Enterprise Nexus) +- **Package manager:** pip / venv standard; `uv` for newer projects +- **Port conventions:** Backend 8000 or 8001; Frontend 3000, 3010, or 5173 + +## Projects Using This Pattern +- [[01 Projects/gmal-scope-builder/GMAL Scope Builder|GMAL Scope Builder]] — FastAPI :8001, PostgreSQL :5433, React :3010 +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — FastAPI :8000, PostgreSQL, React :3000, GCP +- [[01 Projects/video-accessibility/Video Accessibility Platform|Video Accessibility]] — FastAPI + Celery workers, MongoDB, Redis +- [[01 Projects/pdf-accessibility/PDF Accessibility Checker|PDF Accessibility]] — FastAPI + PHP hybrid, PostgreSQL, Redis +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — FastAPI + Celery + Qdrant + PostgreSQL + Redis +- [[01 Projects/olivas/OliVAS|OliVAS]] — FastAPI + DeepGaze + Claude +- [[01 Projects/pimco-charts/PIMCO Chart Generator|PIMCO Charts]] — FastAPI + Claude API +- [[01 Projects/solventum-image-metadata/Solventum Image Metadata|Solventum]] — FastAPI + OpenAI +- [[01 Projects/Oliver-ai-bot_2.0/Oliver AI Bot 2.0|Oliver AI Bot 2.0]] — FastAPI + Next.js + +## Standard Docker Compose Structure +```yaml +services: + backend: + build: ./backend + ports: ["8000:8000"] + env_file: .env + depends_on: [db, redis] + frontend: + build: ./frontend + ports: ["3000:3000"] + db: + image: postgres:16 + ports: ["5432:5432"] + redis: + image: redis:7 +``` + +## Gotchas & Lessons +- GCP Load Balancer has a 30s timeout — long AI calls must use **HTTP polling**, not WebSocket streaming (learned from Mod Comms + Semblance) +- Vite proxy timeout must be increased for long AI operations: `proxy timeout: 300000` (5 min) — learned from GMAL +- `DEV_AUTH_BYPASS` env var pattern for skipping Azure AD auth locally +- For PostgreSQL, always pin port to non-default (e.g., 5433) to avoid conflicts with local installs +- When using Alembic: run `alembic upgrade head` inside the container after first start + +## Related +- [[wiki/tech-patterns/react-vite-typescript|react-vite-typescript]] — paired frontend +- [[wiki/tech-patterns/azure-ad-msal-auth|azure-ad-msal-auth]] — auth layer +- [[wiki/tech-patterns/redis-celery-worker-queue|redis-celery-worker-queue]] — async task processing +- [[wiki/architecture/docker-compose-fullstack|docker-compose-fullstack]] — deployment pattern +- [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]] — GCP gotcha diff --git a/wiki/tech-patterns/nextjs-fastapi-fullstack.md b/wiki/tech-patterns/nextjs-fastapi-fullstack.md new file mode 100644 index 0000000..e9cc23c --- /dev/null +++ b/wiki/tech-patterns/nextjs-fastapi-fullstack.md @@ -0,0 +1,63 @@ +--- +title: "Next.js + FastAPI Fullstack" +aliases: [nextjs-fastapi, nextjs-stack] +tags: [nextjs, fastapi, python, typescript, fullstack] +sources: [01 Projects/enterprise-ai-hub-nexus, 01 Projects/sandbox-notebookllamalm-nextjs, 01 Projects/Oliver-ai-bot_2.0, 01 Projects/ppt-tool] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Next.js + FastAPI Fullstack + +Used for the most complex Oliver AI platforms. Next.js App Router handles SSR + auth; FastAPI handles AI workloads + data. + +## Key Takeaways +- Next.js 14/15 App Router is the choice for production Oliver AI platforms +- Paired with FastAPI backend (Python AI workloads, DB operations) +- Azure AD PKCE auth lives in Next.js layer +- Docker Compose orchestrates all services +- Qdrant for vector search (RAG projects), Redis for caching/queuing + +## When to Use +- Complex, multi-user AI platforms needing SSR, SEO, or M365 integration +- When the frontend state is complex enough to warrant Next.js routing +- Production deployments with multi-service architectures + +## Key Details +- **Next.js versions:** 14 (Enterprise Nexus), 15 App Router (Sandbox NotebookLM) +- **Package manager:** npm (standard), `uv` for Python backend (Sandbox) +- **Auth:** Azure AD PKCE in Next.js; backend validates tokens +- **API pattern:** `src/lib/api.ts` (axios wrapper) → FastAPI backend +- **State:** Zustand (auth store in Sandbox NotebookLM) +- **Real-time:** WebSocket chat routes in FastAPI (Sandbox NotebookLM) + +## Projects Using This Pattern +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise AI Hub Nexus]] — Next.js 14 + FastAPI + PostgreSQL + Redis + Qdrant + Celery +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — Next.js 15 + FastAPI + Python 3.13 + uv + PostgreSQL + Redis + ElevenLabs +- [[01 Projects/Oliver-ai-bot_2.0/Oliver AI Bot 2.0|Oliver AI Bot 2.0]] — Next.js + FastAPI (MVP, 85% backend complete) +- [[01 Projects/ppt-tool/Oliver DeckForge|Oliver DeckForge]] — Next.js :3000 + FastAPI :8000 + nginx :80 + +## Architecture Pattern +``` +Browser → Azure AD PKCE (auth) + ↓ +Next.js (SSR + App Router) + src/lib/api.ts → FastAPI backend + ├── AI service (Claude/Gemini/LlamaIndex) + ├── PostgreSQL (data) + ├── Redis (cache/queue) + └── Celery workers (async jobs) +``` + +## Gotchas & Lessons +- Next.js 15 App Router + React 19 is cutting edge — expect breaking changes in dependencies +- `python-pptx` must be pinned to 0.6.23 for stable PPTX generation (Sandbox NotebookLM) +- `max_tokens` defaults are too low for document generation — increase to 16000+ for long outputs +- nginx reverse proxy needed when Next.js + FastAPI must share a single public port (DeckForge pattern) +- Celery beat for scheduled tasks (SharePoint sync in Enterprise Nexus) needs careful token refresh logic + +## Related +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — backend details +- [[wiki/tech-patterns/azure-ad-msal-auth|azure-ad-msal-auth]] — auth layer +- [[wiki/architecture/rag-architecture|rag-architecture]] — RAG pattern used by Nexus +- [[wiki/tech-patterns/redis-celery-worker-queue|redis-celery-worker-queue]] — async processing diff --git a/wiki/tech-patterns/nodejs-vanilla-proxy.md b/wiki/tech-patterns/nodejs-vanilla-proxy.md new file mode 100644 index 0000000..3ef6dbc --- /dev/null +++ b/wiki/tech-patterns/nodejs-vanilla-proxy.md @@ -0,0 +1,73 @@ +--- +title: "Node.js + Vanilla JS Lightweight Tools" +aliases: [nodejs-proxy, vanilla-js, lightweight-tools] +tags: [nodejs, vanilla-js, proxy, lightweight, no-build] +sources: [01 Projects/3m-portal, 01 Projects/ferrero-ac-creator, 01 Projects/homepage] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Node.js + Vanilla JS Lightweight Tools + +For simple client tools that don't need a frontend build step. Node.js serves static files and acts as a proxy for third-party APIs. + +## Key Takeaways +- Use when: no React needed, no TypeScript, no build pipeline +- `server.js` does two things: serve static HTML/JS/CSS files + proxy `/api/*` to third-party APIs +- No Docker — just `node server.js` or `npm start` +- No transpilation, no bundling — open file in browser or hit localhost directly +- This pattern is for simple client-facing portals and utility tools + +## When to Use +- Wrapping a third-party API that has CORS restrictions +- Small client portals (translation tools, booking tools) +- Rapid prototypes that don't need SSR or TypeScript + +## Key Details + +### server.js Structure +```js +const http = require('http') +const fs = require('fs') +const path = require('path') +const https = require('https') + +const server = http.createServer((req, res) => { + if (req.url.startsWith('/api/')) { + // Proxy to third-party API + proxyRequest(req, res) + } else { + // Serve static files + serveStatic(req, res) + } +}) +server.listen(3000) +``` + +### CORS Proxy Behavior (3M Portal) +- Strips/rewrites `Location` headers on 301/302 → returns 401 (prevents auth redirect loops) +- Injects CORS headers on all `/api` responses +- Masks sensitive data (passwords) in logs + +## Projects Using This Pattern +- [[01 Projects/3m-portal/3M OMG Portal|3M OMG Portal]] — Node.js CORS proxy for One2Edit API (:3000) +- [[01 Projects/ferrero-ac-creator/Ferrero AC Booking Tool|Ferrero AC Booking]] — Node.js server for Box API + CSV download (:3456) +- [[01 Projects/homepage/Homepage Dashboard|Homepage Dashboard]] — Node.js static server for YAML-configured dashboard + +## Comparison: When to Choose What +| Need | Choose | +|------|--------| +| No CORS issues, pure static | Open `index.html` directly (Ferrero "Download CSV" mode) | +| CORS proxy needed | `node server.js` | +| Auth + complex logic | FastAPI (Python) | +| SSR + complex routing | Next.js | + +## Gotchas & Lessons +- `sessionStorage` (not `localStorage`) for auth tokens — clears on browser close (3M Portal) +- Always mask passwords in proxy logs +- 301/302 from API = auth failure in One2Edit — convert to 401 at proxy level +- These tools have no test suite — test manually + +## Related +- [[wiki/tech-patterns/one2edit-api|one2edit-api]] — uses this proxy pattern +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — heavier alternative diff --git a/wiki/tech-patterns/one2edit-api.md b/wiki/tech-patterns/one2edit-api.md new file mode 100644 index 0000000..c7e3877 --- /dev/null +++ b/wiki/tech-patterns/one2edit-api.md @@ -0,0 +1,64 @@ +--- +title: "One2Edit API Integration" +aliases: [one2edit, o2e] +tags: [one2edit, api, translation, 3m, hm] +sources: [01 Projects/3m-portal, 01 Projects/hm-o2e-tool] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# One2Edit API Integration + +One2Edit is an online editor/translation platform used by 3M and H&M for marketing document management. Oliver uses it at `oliver.one2edit.com`. + +## Key Takeaways +- API endpoint: `https://oliver.one2edit.com/v3/Api.php` +- Two auth modes: credential-based (service account) and session-based (externSessionId) +- CORS proxy required — browsers can't call the API directly (same-origin policy) +- The embedded editor uses the One2Edit JS SDK, not REST calls + +## When to Use +Any client project built on the One2Edit platform (3M, H&M). + +## Key Details + +### Two Auth Modes +| Mode | When | How | +|------|------|-----| +| Credential | Fetching job lists | Username → userId, then externSessionId | +| Session | Embedded editor | `externSessionId` in SDK config | + +### 3M Portal Auth Flow +``` +1. Login: username → POST /Api.php → userId +2. Session: userId → POST /Api.php → externSessionId +3. Dashboard: externSessionId → fetch jobs (STARTED/RUNNING) +4. Editor: externSessionId → init One2Edit JS SDK +``` + +### CORS Proxy (3M Portal) +```js +// server.js +// All /api/* requests → oliver.one2edit.com/v3/Api.php +// Strips Location headers on 301/302 → returns 401 +// Injects CORS headers +// Masks passwords in logs +``` + +### Service Account +- 3M Portal uses `portal@oliver.agency` service account for job listing +- Client users get their own `externSessionId` for the embedded editor + +## Projects Using This Pattern +- [[01 Projects/3m-portal/3M OMG Portal|3M OMG Portal]] — Full portal: CORS proxy + Node.js backend + embedded SDK +- [[01 Projects/hm-o2e-tool/HM O2E Tool|H&M O2E Tool]] — Static tool: image relinking + document export (no proxy needed — called directly or via `python -m http.server`) + +## Gotchas & Lessons +- 301/302 redirects from One2Edit mean auth failure — the Node proxy converts them to 401 to prevent redirect loops in the browser +- `sessionStorage` (not `localStorage`) — sessions clear on browser close, which is correct for this auth model +- H&M O2E tool is static (no backend) — can run without a server for most operations +- The JS SDK for the embedded editor is loaded from One2Edit's CDN — needs `externSessionId` at init time + +## Related +- [[wiki/client-knowledge/hm|hm]] — H&M projects +- [[wiki/tech-patterns/nodejs-vanilla-proxy|nodejs-vanilla-proxy]] — CORS proxy pattern diff --git a/wiki/tech-patterns/python-ai-agents.md b/wiki/tech-patterns/python-ai-agents.md new file mode 100644 index 0000000..a1284b7 --- /dev/null +++ b/wiki/tech-patterns/python-ai-agents.md @@ -0,0 +1,99 @@ +--- +title: "Python AI Integration Patterns" +aliases: [ai-agents, claude-api, gemini-api, openai, llm-integration] +tags: [ai, claude, gemini, openai, llamaindex, python, agents] +sources: [01 Projects/gmal-scope-builder, 01 Projects/modcomms, 01 Projects/semblance, 01 Projects/enterprise-ai-hub-nexus, 01 Projects/olivas, 01 Projects/pimco-charts, 01 Projects/video-accessibility, 01 Projects/pdf-accessibility, 01 Projects/build-a-squad, 01 Projects/ac-helper] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Python AI Integration Patterns + +How Oliver projects integrate LLMs — model selection, structured output, multi-model fallback. + +## Key Takeaways +- **Claude** (Anthropic): `tool_use` for structured JSON output — used in GMAL, OliVAS, PIMCO, PDF Accessibility +- **Gemini** (Google): dominant model for real-time/streaming analysis — Mod Comms, Video Accessibility, AC Helper, Build A Squad +- **OpenAI**: used in Semblance (GPT-4.1, GPT-5.2) and Solventum +- **LlamaIndex**: RAG orchestration in Sandbox NotebookLM (multi-model via `llm_factory.py`) +- **DeepGaze**: specialized vision model for OliVAS (saliency/attention prediction) +- Always implement a **fallback model**: Gemini Pro → Flash (Mod Comms pattern) + +## When to Use Which Model + +| Task | Model | Why | +|------|-------|-----| +| Structured JSON output | Claude (`tool_use`) | Reliable schema adherence | +| Document analysis / image | Gemini 2.5 Pro | Multimodal, fast | +| Real-time persona/focus groups | Gemini 3 Pro Preview / GPT-4.1 | Quality personas | +| Metadata generation | OpenAI (Solventum) | Client preference | +| RAG retrieval | LlamaIndex + any model | Framework abstraction | +| Chart/SVG generation | Claude API | Precise code output | +| Prompt optimization | Claude Sonnet 4.6 | Design analysis | +| Video captions | Gemini 2.5 Pro | Native video understanding | + +## Structured Output — Claude `tool_use` +```python +# GMAL pattern: Claude tool_use for schema-bound output +response = client.messages.create( + model="claude-opus-4-6", + tools=[{"name": "parse_brief", "input_schema": BriefSchema}], + tool_choice={"type": "tool", "name": "parse_brief"}, + messages=[{"role": "user", "content": brief_text}] +) +result = response.content[0].input # typed dict +``` + +## Multi-Model Fallback (Mod Comms Pattern) +```python +# Primary: Gemini Pro, fallback: Flash +try: + result = await gemini_pro.generate(prompt) +except Exception: + result = await gemini_flash.generate(prompt) +``` + +## LLM Factory Pattern (Sandbox NotebookLM) +```python +# llm_factory.py — abstract model selection +def get_llm_by_type(llm_type: str) -> BaseLLM: + mapping = {"gemini-pro": GeminiPro(), "gpt4": OpenAI("gpt-4.1"), ...} + return mapping[llm_type] + +def get_structured_llm(schema: type) -> BaseLLM: + ... # wraps output in Pydantic model +``` + +## AI Agents (Mod Comms Multi-Agent) +``` +4 specialist agents (Legal, Brand, Tone, Channel) + → run in parallel + → Lead agent synthesizes verdict +``` +See [[wiki/architecture/multi-agent-ai-systems|multi-agent-ai-systems]] for full pattern. + +## Projects Using This Pattern +- [[01 Projects/gmal-scope-builder/GMAL Scope Builder|GMAL Scope Builder]] — Claude Opus 4.6, tool_use, structured ratecard output +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — Gemini Pro + Flash fallback, 4 parallel agents +- [[01 Projects/semblance/Semblance|Semblance]] — Gemini 3 Pro Preview, GPT-4.1, GPT-5.2, multi-persona +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — multi-model RAG via LlamaIndex + Qdrant +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — LlamaIndex, llm_factory, 7 studio generators +- [[01 Projects/olivas/OliVAS|OliVAS]] — Claude Sonnet 4.6 design analysis + DeepGaze vision +- [[01 Projects/pimco-charts/PIMCO Charts|PIMCO Charts]] — Claude API, SVG code generation +- [[01 Projects/video-accessibility/Video Accessibility Platform|Video Accessibility]] — Gemini 2.5 Pro, VTT captions, audio description +- [[01 Projects/pdf-accessibility/PDF Accessibility Checker|PDF Accessibility]] — Claude + Google Cloud Vision +- [[01 Projects/build-a-squad/Build A Squad|Build A Squad]] — Gemini (client-side, no backend) +- [[01 Projects/ac-helper/AC Helper|AC Helper]] — Gemini, natural language commands + +## Gotchas & Lessons +- Long AI calls (>30s) will timeout on GCP LB — use HTTP polling, not streaming (see [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]]) +- Increase Vite proxy timeout to 5 min for AI-heavy endpoints +- Claude `tool_use` is more reliable than prompt-based JSON for structured output +- `max_tokens` defaults (4096) too low for document generation — set 8192–16000 +- Gemini models change frequently — update model IDs in `llm_factory.py` regularly (Sandbox 2026-03-31) +- GCP Vision + Claude hybrid (PDF Accessibility) — GCV handles image extraction, Claude does semantic analysis + +## Related +- [[wiki/architecture/multi-agent-ai-systems|multi-agent-ai-systems]] +- [[wiki/architecture/rag-architecture|rag-architecture]] +- [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]] diff --git a/wiki/tech-patterns/react-vite-typescript.md b/wiki/tech-patterns/react-vite-typescript.md new file mode 100644 index 0000000..9924374 --- /dev/null +++ b/wiki/tech-patterns/react-vite-typescript.md @@ -0,0 +1,63 @@ +--- +title: "React + Vite + TypeScript Frontend" +aliases: [react-vite, frontend-stack] +tags: [react, vite, typescript, frontend, tailwind, shadcn] +sources: [01 Projects/gmal-scope-builder, 01 Projects/modcomms, 01 Projects/semblance, 01 Projects/video-accessibility, 01 Projects/lusa-back-planner, 01 Projects/build-a-squad, 01 Projects/baic_dashboard] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# React + Vite + TypeScript Frontend + +The standard Oliver frontend stack. React (18 or 19) + Vite + TypeScript, commonly paired with Tailwind CSS and shadcn/ui. + +## Key Takeaways +- React 18 or 19 + Vite is the default for all new SPAs +- Tailwind CSS (CDN for simple tools, npm for complex apps) + shadcn/ui for component-heavy UIs +- `axios` for API calls; `react-router` for routing +- For PDF: `pdfjs-dist` (parse) + `jsPDF` + `html2canvas` (export) +- `shadcn/ui` (Radix UI + Tailwind) used in premium UI tools (Semblance, LUSA Back Planner) +- Recharts for data visualization (BAIC Dashboard) + +## When to Use +Any new Oliver UI that needs interactivity beyond a simple HTML page. + +## Key Details +- **Port conventions:** 3000 (default), 3010 (GMAL), 5137 (Semblance), 5173 (Vite default) +- **State management:** Zustand for auth (Sandbox NotebookLM) or local state +- **API proxy:** Vite proxy to backend — increase timeout for AI calls (5 min) +- **Build:** `npm run build` → `dist/` folder served by nginx or Node +- **MSAL v5:** use `clearCache()` for logout (not `logout()` — breaking change in v5) + +## Styling Variants +| Tool | Styling | +|------|---------| +| Semblance | Tailwind + shadcn/ui | +| LUSA Back Planner | Tailwind + shadcn/ui | +| GMAL Scope Builder | React + Vite + Axios | +| Mod Comms | React 18 + Vite | +| Video Accessibility | React 18 + Vite | +| Build A Squad | React 19 + Vite | +| BAIC Dashboard | React + Vite + Recharts | + +## Projects Using This Pattern +- [[01 Projects/gmal-scope-builder/GMAL Scope Builder|GMAL Scope Builder]] — React 18 + TypeScript + Vite + React Router + Axios +- [[01 Projects/modcomms/Mod Comms|Mod Comms]] — React 18 + Vite + TypeScript +- [[01 Projects/semblance/Semblance|Semblance]] — React + TypeScript + Vite + Tailwind + shadcn/ui +- [[01 Projects/video-accessibility/Video Accessibility Platform|Video Accessibility]] — React 18 + Vite + TypeScript +- [[01 Projects/lusa-back-planner/LUSA Back Planner|LUSA Back Planner]] — React 19 + TypeScript + Vite + Tailwind + shadcn/ui + pdfjs + jsPDF +- [[01 Projects/build-a-squad/Build A Squad|Build A Squad]] — React 19 + TypeScript + Vite + Gemini +- [[01 Projects/baic_dashboard/BAIC Dashboard|BAIC Dashboard]] — React + Vite + Recharts + PapaParse +- [[01 Projects/sandbox-notebookllamalm-nextjs/Sandbox NotebookLM|Sandbox NotebookLM]] — React 19 (inside Next.js 15 App Router) + +## Gotchas & Lessons +- Vite proxy timeout defaults are too low for AI requests — set `proxy: { timeout: 300000 }` in vite.config +- MSAL v5 breaking change: `clearCache()` not `logout()` for sign-out (GMAL fix 2026-03-30) +- Send **ID token** (not Graph access token) to backend for Azure AD validation (GMAL fix 2026-03-30) +- `shadcn/ui` components are copied into `src/components/ui/` — they're not an npm import at runtime +- PapaParse for CSV export is lightweight and works browser-side + +## Related +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — backend counterpart +- [[wiki/tech-patterns/nextjs-fastapi-fullstack|nextjs-fastapi-fullstack]] — Next.js alternative +- [[wiki/tech-patterns/azure-ad-msal-auth|azure-ad-msal-auth]] — auth integration diff --git a/wiki/tech-patterns/redis-celery-worker-queue.md b/wiki/tech-patterns/redis-celery-worker-queue.md new file mode 100644 index 0000000..bb7db4d --- /dev/null +++ b/wiki/tech-patterns/redis-celery-worker-queue.md @@ -0,0 +1,92 @@ +--- +title: "Redis + Celery Async Worker Queue" +aliases: [celery, task-queue, worker, redis-queue] +tags: [redis, celery, async, worker, queue, python] +sources: [01 Projects/enterprise-ai-hub-nexus, 01 Projects/video-accessibility, 01 Projects/pdf-accessibility] +created: 2026-04-15 +updated: 2026-04-15 +--- + +# Redis + Celery Async Worker Queue + +Pattern for offloading long-running AI/processing tasks to background workers. Used in the heaviest Oliver processing pipelines. + +## Key Takeaways +- Redis is both the message broker AND result backend for Celery +- Use Celery when tasks take >5s (AI inference, video processing, PDF analysis) +- `Celery beat` for scheduled recurring tasks (e.g., SharePoint sync) +- PDF Accessibility uses Redis queue directly (`pdf:queue`) without Celery — simpler `worker.py` daemon +- Always poll for task status from the frontend; never block on long tasks + +## When to Use +- Video processing pipelines (multi-phase, minutes-long) +- Scheduled sync jobs (Celery beat) +- Any task that would timeout an HTTP request (>30s) +- Parallel AI analysis tasks + +## Key Details + +### Standard Setup +```yaml +# docker-compose.yml +services: + redis: + image: redis:7 + ports: ["6379:6379"] + worker: + build: ./backend + command: celery -A app.celery worker --loglevel=info + depends_on: [redis] + beat: + build: ./backend + command: celery -A app.celery beat --loglevel=info + depends_on: [redis] +``` + +### Task Definition +```python +@celery.task +def process_video(video_id: str): + # Long-running pipeline + phase_1_ingest(video_id) + phase_2_caption(video_id) # Gemini 2.5 Pro + phase_3_translate(video_id) + phase_4_tts(video_id) +``` + +### Polling Pattern (Frontend) +```js +// Poll until complete +const poll = async (jobId) => { + const { status } = await api.get(`/jobs/${jobId}/status`) + if (status === 'pending') setTimeout(() => poll(jobId), 2000) +} +``` + +## Projects Using This Pattern +- [[01 Projects/enterprise-ai-hub-nexus/Enterprise AI Hub Nexus|Enterprise Nexus]] — Celery beat for SharePoint sync + scheduled tasks; Redis 7 + PostgreSQL +- [[01 Projects/video-accessibility/Video Accessibility Platform|Video Accessibility]] — Celery workers for 7-phase video pipeline; Redis + MongoDB Atlas + GCS +- [[01 Projects/pdf-accessibility/PDF Accessibility Checker|PDF Accessibility]] — Custom `worker.py` daemon reading `pdf:queue` from Redis; PostgreSQL for job tracking + +## Pipeline Phases (Video Accessibility) +``` +1. Upload → Ingestion worker +2. Gemini 2.5 Pro → VTT captions +3. Audio Description generation +4. QC review (approve/reject/edit VTT) +5. Translation → 50+ languages +6. TTS synthesis (GCP TTS + ElevenLabs) +7. Final delivery +``` + +## Gotchas & Lessons +- Celery beat needs its own container — it manages schedules independently from workers +- Proactive token refresh required for long Celery jobs that need M365 access (Enterprise Nexus) +- `worker.py` simpler alternative to Celery for single-queue use (PDF Accessibility pattern) +- Always store job status in DB (not just Redis) so it survives Redis restart +- `video_accessibility_development_plan.txt` is the authoritative spec — always read before touching that pipeline + +## Related +- [[wiki/tech-patterns/fastapi-python-docker|fastapi-python-docker]] — the API layer above +- [[wiki/tech-patterns/python-ai-agents|python-ai-agents]] — what the workers execute +- [[wiki/architecture/gcp-deployment-lb-timeout|gcp-deployment-lb-timeout]] — why polling beats streaming