diff --git a/01 Projects/ac-helper/DEVELOPER_MANUAL.md b/01 Projects/ac-helper/DEVELOPER_MANUAL.md new file mode 100644 index 0000000..6fd641e --- /dev/null +++ b/01 Projects/ac-helper/DEVELOPER_MANUAL.md @@ -0,0 +1,127 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Activation Calendar Helper — Developer Manual + +## Architecture Overview + +The application is a **hybrid stack**: + +1. **Backend**: Python FastAPI application running in Docker. + - Handles API endpoints (`/api/*`), WebSocket connections (`/ws`), and authentication. + - Manages database interactions (PostgreSQL) and AI service integrations. +2. **Frontend**: React application (TypeScript) served via Apache/Nginx as static files. + - Handles UI, state management (Zustand), and MSAL authentication. + - Communicates with the backend via REST API and WebSockets. +3. **Infrastructure**: Docker Compose for local/dev environment orchestration. + - `postgres`: Data persistence. + - `app`: FastAPI container. + +## Tech Stack + +- **Backend**: Python 3.11+, FastAPI, SQLAlchemy, MSAL (Azure AD), OpenAI/Gemini/Claude SDKs. +- **Frontend**: React 18+, TypeScript, Vite, Handsontable (Spreadsheets), MSAL React, Zustand (State). +- **Database**: PostgreSQL 16. +- **Auth**: Azure AD (MSAL PKCE flow). + +## Local Setup + +### Prerequisites +- Docker & Docker Compose. +- Node.js 18+. +- Python 3.11+. + +### Steps + +1. **Clone Repository**: + ```bash + git clone + cd ac-helper + ``` + +2. **Environment Configuration**: + ```bash + cp .env.example .env + cp frontend/.env.example frontend/.env + ``` + - Fill in `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_REDIRECT_URI`. + - Add API keys for desired AI providers (`OPENAI_API_KEY`, `GEMINI_API_KEY`, etc.). + - Set `ADMIN_EMAIL` for initial bootstrap. + +3. **Start Backend**: + ```bash + docker-compose up -d + ``` + - DB will be available at `localhost:5432`. + - API will be available at `http://localhost:${APP_PORT:-8100}`. + +4. **Start Frontend**: + ```bash + cd frontend + npm install + npm run dev + ``` + - Frontend will run on `localhost:5173` and proxy API requests to `localhost:${APP_PORT}`. + +## Environment Variables + +| Variable | Description | Required | +|----------|-------------|----------| +| `AZURE_TENANT_ID` | Azure AD Tenant ID | Yes | +| `AZURE_CLIENT_ID` | Azure AD App Client ID | Yes | +| `AZURE_REDIRECT_URI` | Callback URL | Yes | +| `OPENAI_API_KEY` | OpenAI API Key | Optional | +| `GEMINI_API_KEY` | Google Gemini API Key | Optional | +| `ANTHROPIC_API_KEY` | Anthropic API Key | Optional | +| `EMERGENCY_TOKEN` | Token for SSO bypass | Optional | +| `POSTGRES_PASSWORD` | Database password | Yes | + +## Key Services & Entry Points + +- **`backend/main.py`**: FastAPI app entry point. Registers routers (`/api/auth`, `/api/data`, `/api/ai`, etc.). +- **`frontend/src/main.tsx`**: React app entry point. Initializes MSAL provider and Handsontable modules. +- **`frontend/src/stores/useAuthStore.ts`**: Manages user state and token refresh logic. +- **`frontend/src/api/client.ts`**: Axios instance with interceptors for token injection and error handling. + +## API Reference + +- **`POST /api/auth/login`**: Initiates Microsoft Login. +- **`GET /api/auth/config`**: Returns MSAL configuration for the frontend. +- **`GET /api/data/sheets`**: Lists all sheets. +- **`GET /api/data/sheets/{id}`**: Retrieves sheet data. +- **`POST /api/data/sheets`**: Creates a new sheet. +- **`POST /api/ai/extract`**: Uploads a PDF for AI extraction. +- **`WS /ws`**: WebSocket endpoint for real-time updates. + +## Deployment + +### Production (Apache/Nginx) + +1. **Build Frontend**: + ```bash + cd frontend + npm run build + ``` + Output goes to `frontend/dist`. + +2. **Configure Web Server**: + - Serve `frontend/dist` as static files. + - Proxy `/api/*` and `/ws/*` to the Docker container's `APP_PORT` (default 8100). + +3. **Docker**: + - Set `APP_PORT` in `.env` to match your host port binding. + - Ensure `POSTGRES_PASSWORD` is secure. + - Configure `AZURE_REDIRECT_URI` to the production URL. + +4. **Restart**: `docker-compose up -d`. + +## Known Gotchas + +- **MSAL Redirects**: Ensure `AZURE_REDIRECT_URI` exactly matches the registered Azure App URI. Dev mode requires `http://localhost:5173/ac-helper/` (if proxied) or similar. +- **CORS**: The backend must be configured to allow requests from the frontend origin. +- **File Permissions**: The `data/` volume in Docker must be writable by the app user. +- **AI Rate Limits**: High usage of AI extraction may hit API rate limits. Implement retry logic (already partially implemented via `OPENAI_MAX_RETRIES`). +- **Dependent Dropdowns**: Changing a "Category" or "Media" may invalidate dependent cells. Validate data integrity on import/export. \ No newline at end of file diff --git a/01 Projects/ac-helper/USER_MANUAL.md b/01 Projects/ac-helper/USER_MANUAL.md new file mode 100644 index 0000000..1931ca5 --- /dev/null +++ b/01 Projects/ac-helper/USER_MANUAL.md @@ -0,0 +1,78 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Activation Calendar Helper — User Manual + +## What This Tool Does + +The **Activation Calendar Helper** is a web-based platform for marketing teams to manage deliverables, coordinate campaigns, and extract information from briefs using AI automation. + +### Core Capabilities +- **Smart Spreadsheet**: An interactive grid (powered by Handsontable) for managing deliverables with sorting, filtering, and dependent dropdowns. +- **AI Command Interface**: Natural language commands to create, update, or query deliverables. +- **Brief Extraction**: Upload PDF briefs and use AI (OpenAI, Gemini, or Claude) to automatically populate the spreadsheet with extracted data. +- **Multi-Sheet Management**: Create, rename, duplicate, or delete multiple data sheets. +- **Export**: Export current data to CSV. + +## Who Uses It + +- **Marketing Managers/Coordinators**: To track deliverables across multiple campaigns. +- **Creative Teams**: To upload briefs and manage asset statuses. +- **Admins**: To manage users, dropdown lists (Categories, Media, etc.), and clients. + +## How to Access + +1. **Login**: Navigate to the application URL (e.g., `https://ai-sandbox.oliver.solutions/ac-helper/`). +2. **Microsoft SSO**: Click "Sign in with Microsoft" to authenticate via Azure AD. +3. **Emergency Access**: If SSO is unavailable, click "Emergency access" at the bottom of the login screen and enter the provided token. +4. **Dev Mode**: If running locally in development (`npm run dev`), login is automatic. + +## Main Workflows + +### 1. Managing the Spreadsheet + +The main interface is the spreadsheet grid. + +1. **Navigate Sheets**: Use the sidebar or top bar to switch between sheets. +2. **Create/Duplicate Sheets**: Click the `+` icon or "Duplicate" to create a new dataset. +3. **Edit Data**: Click any cell to edit. Dropdowns are dependent: + - **Category** must be selected first. + - **Media** options depend on the selected Category. + - **Sub-media** options depend on the selected Media. +4. **Sort/Filter**: Use the column headers to sort or apply filters. +5. **Export**: Click "Export CSV" to download the current sheet's data. + +### 2. Processing Briefs (AI Extraction) + +1. **Go to Brief Upload**: Navigate to the "Briefs" section. +2. **Upload File**: Click "Upload Brief" and select a PDF. +3. **Review Extraction**: The AI will parse the document and present the extracted data. +4. **Edit & Confirm**: Verify the data in the review view. Edit any fields if necessary. +5. **Import**: Click "Import to Sheet" to add the deliverables to your active spreadsheet. + +### 3. Using AI Commands + +Use the command input (usually a floating button or top bar) to: +- **Create**: "Create a new deliverable for Category X, Media Y." +- **Update**: "Change the status of Deliverable #123 to 'In Progress'." +- **Query**: "Show me all pending items in the US region." + +## FAQ + +**Q: How do I create new dropdown options (e.g., new Media types)?** +A: Go to **Admin > Dropdowns** (if you have admin rights) to add new categories, media types, or sub-media options. + +**Q: Can I recover a deleted sheet?** +A: No, deletion is permanent. Please ensure you have backups (CSV exports) before deleting. + +**Q: Why is my AI command failing?** +A: Ensure you have the correct API keys configured in the environment. If running locally, check that the AI service keys are valid. + +**Q: How do I export data?** +A: Click the "Export CSV" button in the toolbar. Note: This exports only the *current view* (including any active filters). + +**Q: Is my data secure?** +A: Yes. Data is stored in a PostgreSQL database. Access is restricted to authenticated users. Sensitive files (briefs) are stored securely and not publicly accessible. \ No newline at end of file diff --git a/01 Projects/build-a-squad/DEVELOPER_MANUAL.md b/01 Projects/build-a-squad/DEVELOPER_MANUAL.md new file mode 100644 index 0000000..7f3f543 --- /dev/null +++ b/01 Projects/build-a-squad/DEVELOPER_MANUAL.md @@ -0,0 +1,124 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Build-A-Squad Developer Manual + +## Architecture Overview + +Build-A-Squad is a single-page application (SPA) built with React. It follows a client-side architecture where all state management and calculations occur in the browser. + +### Key Components +1. **Auth Gate:** Manages Microsoft identity integration via MSAL. +2. **App Container:** Orchestrates state between the three main tabs (Scoping, Configurator, Squad). +3. **Data Layer:** Static JSON files for assets and routing logic, parsed at build/initialization time. +4. **AI Service:** Client-side integration with Google Gemini API for scope analysis and auditing. + +## Tech Stack + +- **Framework:** React 19 (using Hooks: `useState`, `useMemo`, `useCallback`). +- **Language:** TypeScript 5.8. +- **Build Tool:** Vite 6. +- **Styling:** Tailwind CSS (via CDN). +- **Auth:** `@azure/msal-react` and `@azure/msal-browser`. +- **AI:** `@google/genai` (Gemini 3.1 Pro). +- **Icons:** Lucide React. + +## Local Setup + +### Prerequisites +- Node.js (LTS version recommended) +- npm or yarn + +### Installation +1. Clone the repository. +2. Install dependencies: + ```bash + npm install + ``` +3. Create a `.env` file in the root directory (see **Environment Variables**). +4. Start the development server: + ```bash + npm run dev + ``` + The app will run at `http://localhost:3000`. + +## Environment Variables + +The application relies on Vite's `define` mechanism to inject environment variables at build time. Create a `.env` file with the following keys: + +```env +# Google Gemini API Key for AI features +GEMINI_API_KEY=your_gemini_api_key_here + +# Microsoft Azure AD B2B/C2B Configuration +AZURE_CLIENT_ID=your_azure_client_id +AZURE_TENANT_ID=your_azure_tenant_id +AZURE_REDIRECT_URI=http://localhost:3000/auth/callback # Or your prod URI +``` + +*Note: Ensure the Redirect URI in the Azure Portal matches the one in your `.env` file.* + +## Key Services & Entry Points + +### `index.tsx` +Entry point. Initializes MSAL, checks authentication status, and renders the `App` component wrapped in `MsalProvider`. + +### `auth.ts` +Configures the `PublicClientApplication` instance for Azure MSAL. Exports `msalInstance` and `loginScopes`. + +### `App.tsx` +The core application logic: +- **State Management:** Manages `selectedAssets`, `overrides`, `manualStaff`, `scenarios`, and tab navigation. +- **Tab Logic:** + - **Scoping:** Calls Gemini API to analyze `scopingInput` and map to `ASSETS_DATA`. + - **Configurator:** Renders filtered lists of `ASSETS_DATA`. + - **Squad:** Calculates FTEs using `STAFFING_ROUTES` and `ROLE_DISCIPLINE_MAP`. +- **Scenario Management:** Saves/loads `Scenario` objects to local state (and supports CSV export/import). + +### `mockData.ts` +Loads static data from JSON files: +- `data/assets.json`: Full catalog of 30+ asset types. +- `data/staffingRoutes.json`: Map of `uniques` to `{ role: hours }`. +- `data/roleDisciplineMap.json`: Map of `role` to `discipline`. + +*Note on Naming:* Despite the filename `mockData.ts`, this serves as the primary data source for the production catalog. The assets are parsed from `documents/Latest Asset List.md` using `scripts/parse-assets.ts`. + +### `scripts/parse-assets.ts` +A utility script to update the catalog data. It parses a pipe-delimited markdown table and outputs the three required JSON files into the `data/` directory. + +Run: `npm run parse-catalog` + +## API Reference + +### AI Integration +The app uses `@google/genai` with `gemini-3.1-pro-preview`. + +**Scope Analysis:** +- **Input:** Project brief text. +- **Output:** JSON schema-based array of `ScopingResult` objects containing `scopeItem`, `volume`, `catalogId`, and `rationale`. +- **Usage:** `GoogleGenAI` instance calls `models.generateContent` with structured output schema. + +**Operational Audit:** +- **Input:** Current `selectedAssets`, `overrides`, `manualStaff`. +- **Output:** Strategic text insights regarding the squad composition. + +## Deployment + +1. **Build:** + ```bash + npm run build + ``` + This generates static files in the `dist/` directory. +2. **Base Path:** In `vite.config.ts`, the base path is set to `/build-a-squad/` for production. Ensure your web server (e.g., IIS, Nginx, Apache) is configured to serve the app from this subdirectory or adjust the `base` config if deploying to the root. +3. **Azure AD:** Update the `AZURE_REDIRECT_URI` in the `.env.production` file to point to your production deployment URL before building for production. + +## Known Gotchas + +1. **State Persistence:** The app does **not** save state to a database. Scenarios are saved only within the current session or via exported CSV. Reloading the page resets all data. +2. **Asset Catalog Updates:** The catalog is static JSON. You **must** run `npm run parse-catalog` after updating `documents/Latest Asset List.md` to see changes in the app. +3. **Billable Hours:** The default FTE calculation uses 1,600 hours/year. Change `billableHoursTarget` in the Squad tab if your agency uses a different divisor (e.g., 2,080). +4. **Tailwind CDN:** The app uses Tailwind via CDN for simplicity in this demo/prototype phase. For production, consider building a full Tailwind CSS pipeline to avoid FOUC (Flash of Unstyled Content) and enable PurgeCSS for smaller bundle sizes. +5. **MSAL Redirect:** If you encounter auth loops, verify that the `redirectUri` in `auth.ts` (and `.env`) exactly matches the one registered in the Azure Portal App Registration under "Authentication". \ No newline at end of file diff --git a/01 Projects/build-a-squad/USER_MANUAL.md b/01 Projects/build-a-squad/USER_MANUAL.md new file mode 100644 index 0000000..4868c61 --- /dev/null +++ b/01 Projects/build-a-squad/USER_MANUAL.md @@ -0,0 +1,85 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Build-A-Squad User Manual + +## What This Tool Does + +Build-A-Squad is a client-side staffing calculator designed for creative agencies. It helps you estimate the team size and composition needed for a project by: +1. **Analyzing your project brief** using AI to suggest relevant creative assets. +2. **Selecting specific deliverables** from a catalog of 30+ asset types. +3. **Calculating FTE (Full-Time Equivalent) projections** by discipline and role based on standardized staffing routes. + +All calculations happen in your browser; no data is sent to a server for storage, though AI features connect to Google Gemini. + +## Who Uses It + +- **Resourcing Managers:** To quickly estimate headcount needs for new proposals. +- **Creative Directors:** To understand the volume and type of creative work involved. +- **Strategy Teams:** To align project scope with budget and resource capabilities. + +## How to Access + +1. Navigate to the application URL provided by your agency. +2. Click the **"Sign in with Microsoft"** button. +3. Authenticate using your Oliver Microsoft account. +4. Once authenticated, you will be directed to the **Scoping** tab. + +## Main Workflows + +The tool follows a three-tab workflow: + +### 1. Scoping (AI-Assisted) +This is the starting point for most projects. + +1. **Paste Brief:** Copy and paste the text of a project brief into the input area. +2. **Analyze:** Click "Analyze Scope." The AI (Google Gemini) will read the brief and suggest relevant assets from the catalog. +3. **Upload Screenshot (Optional):** If you have a PDF or image of the brief, use the upload button to extract text via OCR. +4. **Review Suggestions:** The AI returns a list of suggested assets with rationales. Click the **"Add to Squad"** button on any suggestion to bulk-add it to your selected list. +5. **Adjust:** You can still manually edit volumes or add/remove items later. + +### 2. Configurator (Manual Selection) +Use this tab to browse the full catalog or fine-tune your selections. + +1. **Browse Assets:** View the full list of 30+ asset types. +2. **Filter:** Use the filters to sort by: + - **Category:** (e.g., Brand Communication, Video, Digital). + - **Type:** Master vs. Adapt. + - **Pencil Pro Flag:** Highlighted assets with specific internal flags. +3. **Add Assets:** Click **"Add"** on any asset. A modal will appear where you can set the **Volume** (quantity) if multiple units are needed. +4. **Review Squad:** Switch to the **Squad Projection** tab to see the results of your selections. + +### 3. Squad Projection (FTE Calculation) +This tab displays the calculated staffing requirements. + +1. **View Breakdown:** See the total FTE and hours required, broken down by Discipline (e.g., Creative, Strategy) and specific Roles (e.g., Senior Designer, Planner). +2. **Adjust Parameters:** + - **Billable Hours:** Change the annual billable hours target (default: 1,600) if your agency uses a different standard. + - **Manual Overrides:** Click the pencil icon next to any role's FTE to manually override the calculated number. + - **Bespoke Staff:** Add manual staff entries for roles not in the standard catalog using the "Add Manual Staff" feature. + - **Exclude Roles:** Use the checkmarks to exclude specific roles from the calculation if they are handled by external partners or other departments. +3. **AI Audit:** Click **"Generate AI Audit"** to get strategic insights and recommendations based on your current squad composition. +4. **Save Scenarios:** + - Enter a name in the "Save Scenario" field and click **Save**. + - Select multiple scenarios to compare them side-by-side. Delta indicators will show differences between saved snapshots. +5. **Export:** Use the **Download** button to export your current squad data as a CSV file. + +## FAQ + +**Q: Why do my FTE numbers look different from the project?** +A: The calculator uses standardized "staffing routes" (base hours per asset volume) averaged across the agency. It provides a baseline estimation. Manual overrides are encouraged for specific project nuances. + +**Q: Can I save my work and come back later?** +A: The application does not auto-save to a server. Use the **Save Scenario** feature to create named snapshots of your squad. You can re-load a scenario by pasting the saved data if you have exported it, but note that the primary persistence is within your current browser session via local state or manual save/compare features. + +**Q: How accurate is the AI suggestion?** +A: The AI uses Google Gemini to interpret intent, not just keywords. It is a strong starting point but should always be reviewed. You can manually adjust any suggested asset or volume. + +**Q: Is my project brief data secure?** +A: Yes. The text is sent only to the Google Gemini API for analysis and is not stored by the application itself. No backend database is used. + +**Q: What is the "Pencil Pro" flag?** +A: This indicates assets that have specific internal criteria or flags within the studio's operational model. It is primarily used for internal filtering and reporting. \ No newline at end of file diff --git a/01 Projects/ford_qc/DEVELOPER_MANUAL.md b/01 Projects/ford_qc/DEVELOPER_MANUAL.md new file mode 100644 index 0000000..65cc88d --- /dev/null +++ b/01 Projects/ford_qc/DEVELOPER_MANUAL.md @@ -0,0 +1,181 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Ford BnP QC System Developer Manual + +## Architecture Overview +The system follows a modular architecture: +1. **Entry Points:** + * `qc_engine.py`: Core engine for running QC checks locally or programmatically. + * `ford_qc_box_hotfolder_process.py`: Production daemon that monitors Box folders, orchestrates download, processing, and upload. +2. **Core Logic:** + * `qc_module.py`: Contains `run_qc_checks()` which orchestrates the loading of profiles and execution of check scripts. + * `utils/config.py`: Centralized configuration management, loading from `.env` files. + * `utils/path_resolver.py`: Handles dynamic path resolution for profiles and assets. +3. **Checks Library (`checks/`):** + * Individual Python modules for each QC rule (e.g., `special_requirements_mec_bau.py`). + * `html_reporter.py` / `html_error_reporter.py`: Logic for generating human-readable and error-specific reports. +4. **Configuration:** + * `.env.example`: Template for environment variables. + * `profiles/`: JSON files defining the sequence and parameters of QC checks. + +## Tech Stack +* **Language:** Python 3.8+ +* **Box SDK:** `boxsdk` for cloud storage interaction. +* **PDF Generation:** `reportlab` (used in `generate_pdf.py` for documentation, not core QC). +* **Logging:** `logging` module, with optional `systemd` journal integration. +* **Configuration:** `python-dotenv` (implied by `.env.example` usage). +* **Service Management:** `systemd` for production deployment. + +## Local Setup + +### 1. Clone and Install +```bash +git clone +cd ford_qc_system +python -m venv venv +source venv/bin/activate +pip install -r requirements.txt +``` + +### 2. Environment Variables +Copy `.env.example` to `.env.dev` and configure: + +```ini +FORD_QC_ENV=development +# Box Folders (Can be dummy IDs for local-only testing without Box) +BOX_SOURCE_FOLDER_ID=0 +BOX_REPORT_FOLDER_ID=0 +BOX_PROCESSED_FOLDER_ID=0 +# Optional: Set Box credentials if testing Box integration locally +BOX_CONFIG_FILE=ford_box_config.json +# Paths (relative to script dir) +DOWNLOAD_PATH=download_tmp +WORKING_DIR=working +QC_PROFILE_PATH=profiles/ford_bnp.json +``` + +### 3. Box JWT Config +Place a valid `ford_box_config.json` (from IT or Box Admin) in the project root if testing Box integration. + +## Environment Variables Reference +| Variable | Description | Default/Example | +| :--- | :--- | :--- | +| `FORD_QC_ENV` | Environment mode | `production` or `development` | +| `BOX_SOURCE_FOLDER_ID` | Box folder ID for incoming assets | `332861865120` | +| `BOX_REPORT_FOLDER_ID` | Box folder ID for QC reports | `332864939558` | +| `BOX_PROCESSED_FOLDER_ID` | Box folder ID for processed assets | `332861653811` | +| `BOX_CONFIG_FILE` | Path to Box JWT JSON config | `ford_box_config.json` | +| `DOWNLOAD_PATH` | Local dir for downloads | `download_tmp` | +| `WORKING_DIR` | Local dir for extraction | `working` | +| `QC_PROFILE_PATH` | Path to QC checks definition | `profiles/ford_bnp.json` | +| `BOX_CONNECTION_TIMEOUT` | Box API connect timeout | (see code) | +| `BOX_READ_TIMEOUT` | Box API read timeout | (see code) | +| `MAX_RETRIES` | Max Box API retry attempts | (see code) | +| `WATCHDOG_INTERVAL` | Time between Box folder polls | (see code) | + +## Key Services & Entry Points + +### `run_qc_checks(profile_path, input_file)` +* **Location:** `qc_module.py` +* **Description:** The core function. Loads the QC profile (list of check definitions), iterates through them, and calls the respective check scripts. It aggregates results into a JSON structure. +* **Inputs:** + * `profile_path`: Path to JSON file. + * `input_file`: Path to the `.zip` asset pack. +* **Output:** JSON string containing aggregated results. + +### `run_qc_profile(profile_path, input_file)` +* **Location:** `qc_engine.py` +* **Description:** High-level API for running the profile. Handles JSON loading validation and result aggregation. Used by command-line interface. + +### `ford_qc_box_hotfolder_process.py` +* **Description:** The main production loop. + 1. Authenticates with Box. + 2. Polls `BOX_SOURCE_FOLDER_ID` every `WATCHDOG_INTERVAL` seconds. + 3. Downloads new `.zip` files to `DOWNLOAD_PATH`. + 4. Calls `run_qc_checks` via `qc_module`. + 5. Generates HTML/JSON reports. + 6. Uploads reports to `BOX_REPORT_FOLDER_ID`. + 7. Moves processed `.zip` to `BOX_PROCESSED_FOLDER_ID`. + 8. Handles errors via `HTMLErrorReporter`. + +### `HTMLReporter` and `HTMLErrorReporter` +* **Location:** `checks/html_reporter.py` / `checks/html_error_reporter.py` +* **Description:** Utility classes/functions to format JSON results into Bootstrap-styled HTML. `HTMLErrorReporter` handles specific error types (network, file access, QC failure). + +## API Reference (Internal) + +### QC Profile JSON Structure (`profiles/ford_bnp.json`) +```json +[ + { + "script": "checks.special_requirements_mec_bau", + "config": { + "some_param": "value" + } + }, + ... +] +``` +* `script`: Module path to the check function (e.g., `checks.module_name.check_function`). +* `config`: Dict of parameters passed to the check. + +### Result JSON Structure +```json +{ + "profile": "ford_bnp.json", + "timestamp": "2023-10-27T10:00:00Z", + "checks": [ + { + "script": "checks.special_requirements_mec_bau", + "result": { + "status": "pass", + "details": { "pack_type": "MEC" } + } + } + ] +} +``` + +## Deployment + +### Production (Systemd) +1. Copy `ford_qc_box_hotfolder_process.py` to `/opt/ford_qc/bin/`. +2. Create service file `/etc/systemd/system/ford_qc.service`: + ```ini + [Unit] + Description=Ford BnP QC Box Hotfolder Processor + After=network.target + + [Service] + Type=simple + User=box-cli + Group=box-cli + WorkingDirectory=/opt/ford_qc + EnvironmentFile=/opt/ford_qc/.env.prod + ExecStart=/usr/bin/python3 /opt/ford_qc/bin/ford_qc_box_hotfolder_process.py + Restart=on-failure + # Enable systemd journal logging + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + ``` +3. Enable and start: + ```bash + sudo systemctl daemon-reload + sudo systemctl enable ford_qc.service + sudo systemctl start ford_qc.service + ``` + +## Known Gotchas +1. **Path Resolution:** The script uses `os.path.dirname(os.path.abspath(__file__))` to determine the base directory. Hardcoded paths in `qc_engine.py` (fallback) can cause issues if deployed to a different location. Always rely on `config` or relative paths. +2. **Box Authentication:** JWT keys expire. Ensure the Box JWT key file is rotated regularly. The `BOX_CONFIG_FILE` must be up-to-date. +3. **File Locking:** The system assumes no other process is modifying files in `WORKING_DIR`. It uses basic `os.makedirs` checks but lacks advanced file locking. Ensure exclusive access. +4. **Memory Usage:** Large asset packs can consume significant RAM during extraction. The `FALLBACK_WORKING_DIR` is a safety net if the primary working directory lacks space. +5. **Test Script Specifics:** `test_mec_bau_determination.py` is a debugging helper. Do not run in production. It assumes specific example files exist in `example packs/`. +6. **PDF Generation:** `generate_pdf.py` is for documentation only and has dependencies (`reportlab`, `PIL`, `mmdc`/`mermaid-cli`) not required for the core QC system. \ No newline at end of file diff --git a/01 Projects/ford_qc/USER_MANUAL.md b/01 Projects/ford_qc/USER_MANUAL.md new file mode 100644 index 0000000..d63ac54 --- /dev/null +++ b/01 Projects/ford_qc/USER_MANUAL.md @@ -0,0 +1,121 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# Ford BnP QC System User Manual + +## What This Tool Does +The Ford BnP (Build and Price) QC System is an automated quality control tool designed to validate automotive asset packs before they are deployed to customer-facing applications. It ensures that images, metadata, and configuration files meet strict Ford quality standards. + +**Key Capabilities:** +* **Automated Validation:** Runs 13 specialized checks on asset packs (MEC, BAU, Color Chips, Lifestyle, etc.) to detect errors in image format, metadata correctness, and business rules. +* **Cloud Integration:** Automatically monitors a Box cloud folder for new `.zip` asset packs, downloads them, processes them, and uploads detailed HTML and JSON reports back to a designated Box folder. +* **Detailed Reporting:** Generates rich, Bootstrap-styled HTML reports that clearly list issues found, their severity, and actionable steps to fix them. JSON reports are also generated for programmatic consumption. +* **Error Recovery:** Includes robust error handling for network timeouts, file access issues, and QC failures, with automatic fallback mechanisms. + +## Who Uses It +* **Asset Managers & Creators:** Submitting new vehicle imagery or configuration data and needing confirmation that their uploads are valid. +* **DevOps/IT Operations:** Monitoring the health and status of the asset upload pipeline via logs and Box folder contents. +* **Developers/QA Engineers:** Debugging specific validation failures or adding new QC rules. + +## How to Access +### Production Access (Automated Workflow) +For most users, interaction is indirect. You upload asset packs to a specific Box folder (`BOX_SOURCE_FOLDER_ID`), and the system processes them automatically. + +* **Source Folder (Upload Here):** `332861865120` +* **Reports Folder (View Results Here):** `332864939558` +* **Processed Folder (Archived Files):** `332861653811` + +### Local/Development Access (Manual Workflow) +Developers and QA engineers can run the system locally to test changes or validate specific files. + +**Prerequisites:** +1. Python 3.8+ +2. A valid Ford Box JWT Configuration file (`ford_box_config.json`) +3. The project codebase cloned. + +## Main Workflows (Step-by-Step) + +### Workflow 1: Manual QC Check (Local Development) +Use this to test a specific asset pack file locally. + +1. **Prepare your environment:** + ```bash + # Clone the repository + git clone + cd ford_qc_system + + # Set up virtual environment (recommended) + python -m venv venv + source venv/bin/activate # On Windows: venv\Scripts\activate + pip install -r requirements.txt + ``` + +2. **Create a local configuration:** + ```bash + cp .env.example .env.dev + # Edit .env.dev and ensure FORD_QC_ENV=development + # You may not need Box credentials for local testing if not using Box, + # but BOX_CONFIG_FILE should still point to a valid (or dummy) file if required by imports. + ``` + +3. **Run the QC Engine:** + ```bash + # Basic usage with the default profile + python qc_engine.py profiles/ford_bnp.json --input_file /path/to/your/asset_pack.zip --reports_dir ./local_reports + ``` + +4. **Review Results:** + Check the `./local_reports` directory for `qc_report.html` and `qc_report.json`. + +5. **Generate a specific HTML report from existing JSON (Optional):** + ```bash + python -m checks.html_reporter ./local_reports/qc_report.json + ``` + +### Workflow 2: Monitoring the Box Cloud Folder (Production/Dev Server) +The system runs as a background service to monitor Box. + +1. **Configuration:** + Ensure `.env.prod` is correctly set up with: + * `FORD_QC_ENV=production` + * Correct `BOX_SOURCE_FOLDER_ID`, `BOX_REPORT_FOLDER_ID`, etc. + * Valid `BOX_CONFIG_FILE` path. + +2. **Start the Process:** + ```bash + python ford_qc_box_hotfolder_process.py + ``` + *For systemd deployment, use the provided service file.* + +3. **Check Logs:** + ```bash + journalctl -u ford_qc_system -f + ``` + +4. **Monitor Box:** + * Upload `.zip` files to `BOX_SOURCE_FOLDER_ID`. + * Wait for processing. + * Check `BOX_REPORT_FOLDER_ID` for new HTML/JSON reports. + * Original `.zip` files are moved to `BOX_PROCESSED_FOLDER_ID`. + +## FAQ + +**Q: Where do I find the list of QC checks?** +A: The checks are defined in `profiles/ford_bnp.json`. Each entry specifies a script and configuration. + +**Q: How do I fix a failed QC check?** +A: Open the generated HTML report in `BOX_REPORT_FOLDER_ID` or your local `reports_dir`. The report provides specific details and "Fix Instructions" for each error. + +**Q: What if the Box API connection fails?** +A: The system includes retry logic (`MAX_RETRIES`, `RETRY_BACKOFF_BASE`). If it exceeds retries, it logs the error and moves the file to an error state (often a specific error report folder or leaving it in place with a logged error) depending on configuration. Check logs for details. + +**Q: Can I add custom QC checks?** +A: Yes. The system is modular. Create a new Python script in the `checks/` directory, define it in `profiles/ford_bnp.json` with the appropriate `script` name and `config` parameters, and restart the service. + +**Q: Where are the logs stored?** +A: +* **Production:** Systemd Journal (`journalctl -u ford_qc_system`) +* **Local/Dev:** Standard output or log files configured in the logging setup section of `ford_qc_box_hotfolder_process.py`. diff --git a/01 Projects/gmal-scope-builder/DEVELOPER_MANUAL.md b/01 Projects/gmal-scope-builder/DEVELOPER_MANUAL.md new file mode 100644 index 0000000..33a4280 --- /dev/null +++ b/01 Projects/gmal-scope-builder/DEVELOPER_MANUAL.md @@ -0,0 +1,163 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# GMAL Scope Builder Developer Manual + +## Architecture Overview + +The application is a Dockerized, three-service architecture: +1. **Frontend**: React + Vite + TypeScript SPA. +2. **Backend**: Python FastAPI application handling business logic, AI integration, and database interactions. +3. **Database**: PostgreSQL 16 storing project data, GMAL assets, and user information. + +Services communicate via HTTP (Frontend <-> Backend) and network (Backend <-> Database). + +## Tech Stack + +- **Containerization**: Docker & Docker Compose +- **Database**: PostgreSQL 16 (Alpine image) +- **Backend**: Python, FastAPI, `asyncpg` (async driver), `psycopg2` (sync driver) +- **Frontend**: React 18+, Vite, TypeScript, `react-router-dom`, `@azure/msal-react` (Authentication) +- **AI**: Anthropic Claude API (Opus 4.6) +- **Auth**: Microsoft Entra ID (Azure AD) via MSAL + +## Local Setup + +### Prerequisites +- Docker & Docker Compose installed. +- Git. +- An Active Anthropic API Key. + +### Steps + +1. **Clone & Setup Env**: + ```bash + git clone git@bitbucket.org:zlalani/gmal-scope-builder.git + cd gmal-scope-builder + cp .env.example .env + nano .env + ``` + - Set `ANTHROPIC_API_KEY`. + - Set `POSTGRES_PASSWORD`. + - For local dev, set `DEV_AUTH_BYPASS=true` and `VITE_DEV_AUTH_BYPASS=true` to skip Azure login. + +2. **Add Data**: + Place the master GMAL Excel file (e.g., `U-Studio GMAL Asset Job Routes Apr25 ForFranky.xlsx`) in the `data/` directory. + +3. **Build & Run**: + ```bash + docker compose build + docker compose up -d + ``` + +4. **Wait for Services**: + Wait for PostgreSQL to be healthy and Backend to startup. + ```bash + docker compose logs backend --tail 5 + ``` + +5. **Ingest GMAL Data**: + ```bash + curl -X POST http://localhost:8001/api/gmal/ingest + ``` + +6. **Access App**: + Open `http://localhost:3010`. + +## Environment Variables + +### `.env` (Root) +- `POSTGRES_PASSWORD`: Password for the `scope_user` DB user. +- `ANTHROPIC_API_KEY`: Key for Claude API access. +- `AZURE_TENANT_ID` & `AZURE_CLIENT_ID`: For MSAL authentication. +- `DEV_AUTH_BYPASS`: Set to `true` to disable SSO in local dev. +- `VITE_DEV_AUTH_BYPASS`: Frontend flag to bypass auth UI. +- `DATA_DIR`: Absolute path to directory containing the GMAL Excel file. + +### `docker-compose.yml` (Backend Service) +- `DATABASE_URL`: Async SQLAlchemy connection string. +- `DATABASE_URL_SYNC`: Sync SQLAlchemy connection string. +- Environment variables are passed through from the root `.env`. + +## Key Services & Entry Points + +### Backend +- **Entry Point**: Implicit in `./backend` Docker build context (likely `main.py` or `app.py` not shown, but standard FastAPI structure). +- **GMAL Ingestion**: Endpoint `POST /api/gmal/ingest`. Reads Excel from `DATA_DIR`, parses assets/roles/hours, and populates the DB. +- **AI Integration**: Uses `ANTHROPIC_API_KEY`. Exposes `/ai/usage` and `/ai/debug` endpoints for frontend tracking. +- **Projects**: CRUD endpoints at `/projects`. + +### Frontend +- **Entry Point**: `frontend/src/main.tsx` -> `App.tsx`. +- **Auth**: `frontend/src/auth/AuthProvider.tsx`. + - Uses `@azure/msal-react`. + - Checks `import.meta.env.VITE_DEV_AUTH_BYPASS` to render raw children if true. +- **Routing** (`frontend/src/App.tsx`): + - `/`: Dashboard (List Projects). + - `/new`: NewProject Form. + - `/projects/:id`: ProjectView. + - `/gmal`: GMAL Browser. + - `/gmal-editor`: GMAL Editor (Admin only). + - `/users`: UserManagement (Admin only). + - `/help`: Help Page. +- **Key Components**: + - `Dashboard.tsx`: Displays stats and project list. + - `NewProject.tsx`: Form for creating scopes. + - `AiCostTracker.tsx`: Polls `/ai/usage` every 5s. + - `DebugPanel.tsx`: Polls `/ai/debug` every 3s for LLM debug info. + +## API Reference (Key Endpoints) + +| Method | Path | Description | Auth | +| :--- | :--- | :--- | :--- | +| POST | `/api/projects` | Create a new project scope | Auth | +| GET | `/api/projects` | List all projects | Auth | +| POST | `/api/gmal/ingest` | Re-ingest GMAL Excel file | Admin | +| GET | `/api/gmal/stats` | Get GMAL asset/role counts | Auth | +| GET | `/api/ai/usage` | Get cumulative AI cost/token usage | Internal/Dev | +| GET | `/api/ai/debug` | Get recent LLM debug logs | Internal/Dev | + +## Deployment + +### On a Remote Server + +1. **Copy Code & Data**: + ```bash + git clone git@bitbucket.org:zlalani/gmal-scope-builder.git + cd gmal-scope-builder + # Copy Excel file and DB backup if restoring + scp "data/*.xlsx" user@server:/path/to/gmal-scope-builder/data/ + ``` + +2. **Configure `.env`**: + ```bash + cp .env.example .env + nano .env + ``` + Ensure production values for passwords and keys. **Never** set `DEV_AUTH_BYPASS=true` in production. + +3. **Adjust Ports**: + Edit `docker-compose.yml` ports if `127.0.0.1:8002:8000`, `5433`, and `3010` conflict with existing services. + +4. **Start**: + ```bash + docker compose build + docker compose up -d + ``` + +5. **Initial Ingest**: + ```bash + curl -X POST http://localhost:8001/api/gmal/ingest # Or your mapped port + ``` + +## Known Gotchas + +1. **Double-Counting Efficiency**: When using "AI efficiency" model types in the frontend, do not manually apply additional AI efficiency adjustments. The model type already accounts for this. +2. **Data Directory Path**: The backend mounts `${DATA_DIR:-./data}`. Ensure this directory exists and is writable by the container user, and contains the valid Excel file. Relative paths in `.env` are resolved relative to the repo root. +3. **Auth Bypass**: `DEV_AUTH_BYPASS` and `VITE_DEV_AUTH_BYPASS` must remain empty/false in production to ensure security. +4. **AI Cost**: The AI costs can accumulate quickly during development/debugging due to frequent calls. Monitor the `AiCostTracker` widget. +5. **Database Restore**: If restoring from `scope_builder_deploy.sql`, ensure the DB user and database name (`scope_builder`, `scope_user`) match the environment variables. +6. **Excel Format**: The GMAL ingestion logic depends on the specific structure of the "U-Studio GMAL Asset Job Routes..." Excel file. Changes to this file structure will break ingestion unless the backend parser is updated. \ No newline at end of file diff --git a/01 Projects/gmal-scope-builder/USER_MANUAL.md b/01 Projects/gmal-scope-builder/USER_MANUAL.md new file mode 100644 index 0000000..973841f --- /dev/null +++ b/01 Projects/gmal-scope-builder/USER_MANUAL.md @@ -0,0 +1,86 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# GMAL Scope Builder User Manual + +## What This Tool Does + +The GMAL Scope Builder is a web application designed for agencies to efficiently scope new client ratecards and team structures. It works by matching client briefs against a standardized master asset database (GMAL) using AI (Claude Opus 4.6). + +Key capabilities include: +- **Automated Asset Matching**: AI matches client requirements to standardized GMAL assets. +- **Ratecard Building**: Automatically calculates hours per role based on matched assets. +- **Team Shaping**: Calculates team Full-Time Equivalent (FTE) headcount. +- **Cost Tracking**: Monitors AI usage costs and token consumption. + +## Who Uses It + +- **Project Managers/Scopers**: Primary users who create new projects, input client briefs, and review AI-generated scopes. +- **Administrators**: Users with 'admin' role who can manage GMAL data (ingest new master files), edit the GMAL database, and manage user accounts. +- **Viewers**: Users who can view existing projects and GMAL assets but cannot create new scopes or modify data. + +## How to Access + +1. Navigate to the application URL in your web browser. +2. **Authentication**: You will be redirected to the Microsoft login page. Sign in with your Oliver Agency Microsoft account. + - *Note*: Local development environments can bypass SSO by setting `VITE_DEV_AUTH_BYPASS=true` in the `.env` file. +3. Once logged in, you will see the **Dashboard** (Projects list). + +## Main Workflows + +### Workflow 1: Creating a New Project Scope + +1. **Navigate**: Click the **"+ New Project"** button on the Dashboard (available to Editors/Admins). +2. **Enter Details**: + - **Project Name**: e.g., "Acme Corp Q2 2025 Scope". + - **Client Name**: The client's name. + - **Description**: Paste or type the client brief/requirements. + - **Model Type**: Select the appropriate AI model type from the dropdown. + - *Tip*: If selecting an "AI efficiency" model type, do not apply additional AI efficiency profiles to avoid double-counting hours. +3. **Submit**: Click **"Create Project"**. You will be redirected to the **Project View** page. +4. **Review**: The AI will process the brief. Review the matched assets, suggested hours, and team composition. + +### Workflow 2: Viewing Projects + +1. **Dashboard**: View all your projects with their current status (Draft, Parsing, Matching, Review, Building, Finalized). +2. **Status Indicators**: + - Green: Finalized + - Blue: Matching/Parsing + - Yellow: Draft/Building + - Grey: Draft (Muted) +3. **Click a Project**: Opens the detailed **Project View** to inspect or edit the scope. + +### Workflow 3: Managing GMAL Data (Admins Only) + +1. **Ingesting New Data**: + - Click **"Ingest GMAL Data"** on the Dashboard. + - Confirm the action. This reloads all assets from the master Excel file. + - *Warning*: This process updates the entire database. Ensure the correct Excel file is in the `data/` directory. +2. **Editing GMAL Assets**: + - Navigate to **"GMAL Editor"** (Admin only). + - Use the interface to search, view, and modify assets, roles, and hourly records in the GMAL master database. + +### Workflow 4: Browsing Assets + +1. Navigate to **"GMAL Browser"** from the sidebar. +2. Search the standardized asset database to understand available resources and standard rates. + +## FAQ + +**Q: Why does the AI cost tracker show increasing costs?** +A: The app uses Claude Opus 4.6 for AI matching. Costs accumulate based on input/output tokens and the number of API calls. You can monitor this in real-time via the "AI Cost" widget in the app. + +**Q: Can I edit the client brief after creation?** +A: Yes, on the Project View page, you can modify the description or other parameters and re-run the AI matching process. + +**Q: What is the difference between "Model Type" options?** +A: Different model types account for different levels of AI efficiency in the hourly rates. Some models have AI efficiency built-in; others do not. Selecting the correct type prevents double-counting efficiency gains. + +**Q: I don't see the "Admin" menu items.** +A: You need the 'admin' role assigned to your user account. Contact your system administrator if you believe you should have access. + +**Q: How is FTE calculated?** +A: FTE is calculated based on the total hours required for all matched assets, divided by the standard annual working hours for a full-time employee. diff --git a/01 Projects/olivas/DEVELOPER_MANUAL.md b/01 Projects/olivas/DEVELOPER_MANUAL.md new file mode 100644 index 0000000..55ef428 --- /dev/null +++ b/01 Projects/olivas/DEVELOPER_MANUAL.md @@ -0,0 +1,124 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# OliVAS Developer Manual + +## Architecture Overview +OliVAS is a full-stack web application with a clear separation between frontend and backend services, managed via Docker Compose. + +- **Frontend**: React 18 SPA with TypeScript, Vite, Tailwind CSS, Zustand (state), and React Router. +- **Backend**: FastAPI (Python) with async SQLAlchemy and Pydantic v2. Handles API logic, file processing, and ML model inference. +- **Database**: PostgreSQL 16 (via Docker). +- **Auth**: Azure AD SSO via MSAL. +- **ML**: Local GPU/CPU inference or offloaded to Google Cloud Run. + +## Tech Stack +| Layer | Technology | +|-------|------------| +| **Frontend** | React 18, TypeScript, Vite, Tailwind CSS, Zustand, React Router | +| **Backend** | FastAPI, Python 3.12, SQLAlchemy (async), Pydantic v2 | +| **Database** | PostgreSQL 16 | +| **Auth** | Azure AD (MSAL) | +| **ML** | DeepGaze Models, Optional: Anthropic API | + +## Local Setup + +### Prerequisites +- Docker and Docker Compose +- Python 3.12+ (for local backend dev if not using container) + +### Step-by-Step +1. **Clone the Repository** + ```bash + git clone + cd olivas + ``` + +2. **Configure Environment** + Copy `.env.example` to `.env`: + ```bash + cp .env.example .env + ``` + +3. **Start Services** + ```bash + docker-compose up --build + ``` + This starts: + - `postgres` on port `5453` + - `backend` on port `8000` + - `frontend` on port `1577` + +4. **Access the App** + - Frontend: `http://localhost:1577/olivas` + - Backend API: `http://localhost:8000/docs` + +## Environment Variables + +### General +- `DATABASE_URL`: PostgreSQL connection string (e.g., `postgresql+asyncpg://olivas:olivas@localhost:5453/olivas`) +- `UPLOAD_DIR`: Local path for storing uploaded images (e.g., `./data/uploads`) +- `DEVICE`: ML device (`auto`, `cpu`, `cuda`) +- `CORS_ORIGINS`: Allowed frontend origins (e.g., `http://localhost:1577`) +- `BACKEND_HOST/PORT`: Server binding settings + +### Authentication (Azure AD) +- `AZURE_AUTH_ENABLED`: `true`/`false` +- `AZURE_TENANT_ID`: Your Azure AD Tenant ID +- `AZURE_CLIENT_ID`: Your Azure AD Application Client ID +- `VITE_AZURE_*`: Frontend Azure AD config (Tenant, Client, Redirect URI) + +### Optional Services +- `ANTHROPIC_API_KEY`: Enable AI Design Analysis via Claude Sonnet 4.6 +- `CLOUD_RUN_*`: Enable Google Cloud Run offloading for ML models + +## Key Services & Entry Points + +### Frontend (`frontend/src/`) +- **`App.tsx`**: Route definitions. All routes are protected by `RequireAuth`. +- **`auth/RequireAuth.tsx`**: Checks MSAL authentication status. Redirects to Login if unauthenticated. +- **`auth/msalConfig.ts`**: MSAL instance configuration. +- **`components/AuthImage.tsx`**: Helper to fetch images via authenticated API requests and convert to Blob URLs. + +### Backend (`backend/`) +- **Main Entry**: FastAPI app initialization (see `main.py` or `app.py` in backend dir). +- **API Client**: Frontend uses a configured axios/fetch client with baseURL `/api`. +- **ML Inference**: + - Local: Uses DeepGaze models based on `DEVICE` env var. + - Cloud: Sends requests to `CLOUD_RUN_SALIENCY_URL` if configured. + +## API Reference + +### Authentication +- **Login**: Azure AD Login Redirect. Handled by MSAL. +- **Protected Endpoints**: All API endpoints require a valid Azure AD bearer token. + +### Endpoints (Examples) +- `GET /api/projects`: List all projects. +- `POST /api/projects`: Create a project. +- `POST /api/analyses`: Trigger analysis on an uploaded image. +- `GET /api/analyses/{id}`: Get analysis details (heatmap URLs, metrics, insights). +- `GET /api/analyses/{id}/image`: Retrieve original image (authenticated blob). + +## Deployment + +### Production Considerations +1. **HTTPS**: Ensure reverse proxy (Nginx/Traefik) handles SSL termination. +2. **CORS**: Update `CORS_ORIGINS` to include production domain. +3. **Secrets**: Do not commit `.env` files. Use CI/CD secrets or Vault. +4. **Storage**: Map `UPLOAD_DIR` to persistent cloud storage (e.g., S3) in production. +5. **ML Acceleration**: Set `DEVICE=cuda` and ensure GPU drivers are available on the host. + +### Google Cloud Run Offload +- Set `CLOUD_RUN_SALIENCY_URL` and `CLOUD_RUN_PROCESSING_URL` to point to deployed Cloud Run services. +- Set `GOOGLE_CLOUD_PROJECT`. + +## Known Gotchas +1. **MSAL Redirect Issues**: Ensure `VITE_AZURE_REDIRECT_URI` matches the Azure AD portal configuration exactly. Note the `/olivas` suffix for the frontend basename. +2. **Image Loading**: Images stored on the server must be fetched via `/api/analyses/{id}/image` to include auth tokens. Do not use direct filesystem paths in production. +3. **Database Connection**: The backend uses asyncpg. Ensure `DATABASE_URL` uses the `postgresql+asyncpg://` dialect. +4. **CORS**: During development, ensure `CORS_ORIGINS` includes `http://localhost:1577`. In production, specify your actual domain. +5. **AI Insights Disabled**: If `ANTHROPIC_API_KEY` is empty, the AI Insights tab will show a message indicating AI analysis is unavailable. \ No newline at end of file diff --git a/01 Projects/olivas/USER_MANUAL.md b/01 Projects/olivas/USER_MANUAL.md new file mode 100644 index 0000000..01ff556 --- /dev/null +++ b/01 Projects/olivas/USER_MANUAL.md @@ -0,0 +1,102 @@ +--- +auto_generated: true +manual_updated_at: 2026-05-18 +modified: 2026-05-18 +--- + +# OliVAS User Manual + +## What This Tool Does +OliVAS (OLIVER Visual Attention Suite) is an open-source web application designed for creative teams, designers, and marketers. It predicts where human eyes will look in an image during the first 3-5 seconds of viewing. + +Instead of using physical eye-tracking hardware, OliVAS uses advanced machine learning models to generate: +- **Saliency Heatmaps**: Visual overlays showing predicted attention intensity. +- **Gaze Sequence Predictions**: Numbered points showing the predicted order of scanning. +- **Hotspot Detection**: Identification of the top 5 attention regions. +- **AI Design Analysis**: Optional insights powered by Claude Sonnet 4.6, referencing specific visual elements to provide actionable recommendations. + +## Who Uses It +- **Designers**: To validate layout hierarchy and focal points. +- **Marketers**: To optimize ad creatives for maximum engagement. +- **Creative Directors**: To provide data-driven feedback on visual assets. + +## How to Access +1. Navigate to the OliVAS application URL. +2. Click **"Sign in with Microsoft"**. +3. If you do not have an Oliver agency account, access will be denied. Contact your IT administrator if you believe you should have access. + +## Main Workflows + +### 1. Creating a Project +1. Log in to the Dashboard. +2. Click **"Create Project"** in the top right corner. +3. Enter a **Project Name** and optional **Description**. +4. Click **"Create"**. +5. You will be redirected to the Project Detail page. + +### 2. Analyzing an Image +1. From the **Dashboard** or **Project Detail** page, click **"Analyze"** or **"Create New Analysis"**. +2. Upload an image file. +3. Select a **Model**: + - **DeepGaze I**: Fast, baseline saliency. + - **DeepGaze IIE**: **Recommended** for best accuracy. + - **DeepGaze III**: High-resolution detail. +4. Click **"Analyze"**. +5. Wait for the analysis to complete. You will be redirected to the **Analysis View**. + +### 3. Interpreting Results (Analysis View) +Once on the Analysis View page, you can explore several tabs: + +#### Heatmap Tab +- Toggle **Opacity** to adjust the visibility of the saliency heatmap overlay. +- Select a **Colormap** (Jet, Viridis, Inferno, etc.) to change how intensity is displayed. +- The red/hot areas indicate high predicted attention. + +#### Gaze Sequence Tab +- View **numbered fixation points** (1, 2, 3...) showing the predicted order viewers will scan the image. +- Use this to understand the visual path created by your design. + +#### Hotspots Tab +- See the **top 5 attention regions** ranked by intensity. +- Bounding boxes highlight these key areas. + +#### AOI (Areas of Interest) +- Draw rectangles over specific design elements. +- The tool calculates **Attention %** (how much attention the element gets), **Area %** (size relative to image), and **Attention Density** for the selected area. + +#### Insights Tab +- **Rule-Based Insights**: Automatic metrics including: + - **Attention Score**: 0-100 concentration score. + - **Focal Dominance**: How much one area dominates the view. + - **Spatial Balance**: Distribution of attention across the image. + - **Edge Risk**: Potential for attention drifting off the edges. +- **AI Design Analysis**: If enabled, view text-based insights from Claude Sonnet 4.6. These reference specific visual elements and provide actionable design recommendations. + +#### PDF Report +- Click **"Download PDF Report"** to generate a professional document containing: + - All visualizations (heatmaps, gaze sequences). + - Metrics (Hotspots, AOI data). + - Rule-based and AI insights. + - Montserrat typography for readability. + +### 4. Comparing Designs +1. Navigate to two different analyses. +2. Use the **Comparison View** feature. +3. View side-by-side metrics and heatmaps to evaluate performance differences. + +## FAQ + +**Q: Why is my analysis taking a long time?** +A: Depending on the model selected (DeepGaze IIE/III) and server load, analysis may take several seconds. DeepGaze IIE is recommended for a balance of speed and accuracy. + +**Q: Can I use OliVAS for external client work?** +A: Yes, if you have an Oliver agency account. Ensure you comply with your organization's data privacy policies regarding client images. + +**Q: What is the "Attention Score"?** +A: It is a 0-100 metric measuring how focused the predicted attention is. A higher score (e.g., >55) indicates a strong focal point, while a lower score suggests a diffuse or scattered gaze pattern. + +**Q: How do I get AI Insights?** +A: AI Insights are optional and require an Anthropic API key to be configured in the backend. If available, they will appear in the Insights tab. If not, only rule-based insights will be shown. + +**Q: Can I delete an analysis?** +A: Yes, from the Analysis View or Project Detail page, use the **"Delete"** button. This action is usually irreversible. diff --git a/wiki/_master-index.md b/wiki/_master-index.md index 01b1438..7c3c4e4 100644 --- a/wiki/_master-index.md +++ b/wiki/_master-index.md @@ -23,7 +23,7 @@ This 3-hop pattern works for hundreds of articles without vector search. | [[wiki/tech-patterns/_index\|tech-patterns/]] | Recurring tech stacks: FastAPI, React/Vite, Next.js, Azure AD, AI, Box, One2Edit, Redis/Celery, cost-tracker, OMG API, Payload CMS | 39 | | [[wiki/architecture/_index\|architecture/]] | Cross-cutting architectural patterns: Docker Compose, multi-agent AI, GCP timeout, RAG, hotfolder, optical-dev deploy, cost-tracker, new-project checklist, troubleshooting playbooks, ADR log, Cloud Run Jobs | 53 | | [[wiki/client-knowledge/_index\|client-knowledge/]] | Per-client notes for Ford, H&M, L'Oréal, Barclays, Ferrero, 3M, BAIC | 7 | -| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 202 | +| [[wiki/concepts/_index\|concepts/]] | Atomic knowledge extracted from Claude Code sessions | 204 | | [[wiki/connections/_index\|connections/]] | Cross-cutting insights linking 2+ concepts: FastAPI+Azure AD+Docker trinity, AI→cost-tracker, Apache+Vite basePath, GCP→REST polling, Box+hotfolder, Docker DNS+AdGuard, Celery prefork×faster_whisper memory stacking | 10 | | [[wiki/qa/_index\|qa/]] | Filed answers to queries (saved with `--file-back`) | 0 | | [[wiki/homelab/_index\|homelab/]] | Self-hosted infra: Proxmox install, IOMMU/PCI passthrough, hypervisor setup, budget builds, HP Elitedesk G3, Homarr API + Apps + Boards + Certificates + Integrations + Settings + Tasks + AdGuard + Clock + Docker Stats + Docker Integration + Download Client + Firewall + Proxmox Integration + Radarr + Readarr + Sonarr + Bookmarks + Calendar + Icons + App Widget + Weather + GitHub + Nextcloud + qBittorrent + RSS Feed + Speedtest Tracker + System Health Monitoring + System Resources + Services Map + Media Stack | 43 | diff --git a/wiki/concepts/_index.md b/wiki/concepts/_index.md index b14d952..a623c84 100644 --- a/wiki/concepts/_index.md +++ b/wiki/concepts/_index.md @@ -2,14 +2,14 @@ title: "Concepts Index" description: "Atomic knowledge extracted from Claude Code sessions — gotchas, bugs, workarounds" tags: [index, concepts] -updated: 2026-05-15 +updated: 2026-05-18 --- # Concepts — Topic Index Atomic knowledge articles from real Oliver Agency sessions. Each article = one specific gotcha, bug, or non-obvious behaviour discovered in production or testing. -**202 articles** — use Obsidian search or `grep` to find by keyword. +**204 articles** — use Obsidian search or `grep` to find by keyword. | Article | Title | Created | |---------|-------|---------| @@ -39,6 +39,7 @@ Atomic knowledge articles from real Oliver Agency sessions. Each article = one s | [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error\|celery-redis-queue-flush-on-deterministic-error]] | Celery + Redis: Must Flush Redis on Deterministic Errors | 2026-04-30 | | [[wiki/concepts/celery-worker-restart-inflight-task-loss\|celery-worker-restart-inflight-task-loss]] | Celery Worker Restart Orphans In-Flight Tasks | 2026-05-08 | | [[wiki/concepts/chartjs-time-axis-adapter\|chartjs-time-axis-adapter]] | Chart.js — type:time Axis Requires a Date Adapter | 2026-04-21 | +| [[wiki/concepts/chromium-pdf-css-font-injection\|chromium-pdf-css-font-injection]] | Chromium PDF — @font-face Must Be in HTML for CIDFont Embedding | 2026-05-18 | | [[wiki/concepts/claude-code-plugin-marketplace\|claude-code-plugin-marketplace]] | Claude Code Plugin Marketplace Install Pattern | 2026-04-29 | | [[wiki/concepts/claude-code-schedule-skill-account-type\|claude-code-schedule-skill-account-type]] | Claude Code — /schedule Skill Requires claude.ai Login Account | 2026-04-24 | | [[wiki/concepts/cline-lm-studio-openai-compatible\|cline-lm-studio-openai-compatible]] | Cline + LM Studio: Use OpenAI Compatible Provider | 2026-04-30 | @@ -179,6 +180,7 @@ Atomic knowledge articles from real Oliver Agency sessions. Each article = one s | [[wiki/concepts/pydantic-v2-alias-id-gotcha\|pydantic-v2-alias-id-gotcha]] | Pydantic v2 — Field(alias='_id') Serializes as _id, Breaking Frontend .id Access | 2026-04-27 | | [[wiki/concepts/pydub-ffmpeg-silent-dependency\|pydub-ffmpeg-silent-dependency]] | pydub Silent ffmpeg Dependency in Docker | 2026-04-30 | | [[wiki/concepts/python-fastapi-module-level-singletons\|python-fastapi-module-level-singletons]] | Module-Level Singletons Break pytest — Use Lazy Initialisation | 2026-04-30 | +| [[wiki/concepts/python-float-formatting-comparison\|python-float-formatting-comparison]] | Python Float — Equality Trap When Formatting Percentages (1.1×100≠110) | 2026-05-18 | | [[wiki/concepts/python-iso-z-suffix\|python-iso-z-suffix]] | Python — fromisoformat() Cannot Parse Z Suffix (Python < 3.11) | 2026-04-24 | | [[wiki/concepts/python-service-deployment-dotenv\|python-service-deployment-dotenv]] | Python Service Deployment — venv and .env Checklist | 2026-04-16 | | [[wiki/concepts/qbittorrent-proxy-profile-blocking\|qbittorrent-proxy-profile-blocking]] | qBittorrent Proxy Profile Enabled Without Proxy Host Blocks All Connections | 2026-05-03 | diff --git a/wiki/concepts/chromium-pdf-css-font-injection.md b/wiki/concepts/chromium-pdf-css-font-injection.md new file mode 100644 index 0000000..73d72d7 --- /dev/null +++ b/wiki/concepts/chromium-pdf-css-font-injection.md @@ -0,0 +1,78 @@ +--- +title: "Chromium PDF — CSS Font-Face Must Be in HTML , Not SVG " +tags: [chromium, pdf, css, fonts, svg, inkscape, indesign] +created: 2026-05-18 +sources: daily/2026-05-18.md +--- + +# Chromium PDF — CSS Font-Face Must Be in HTML ``, Not SVG `` + +## The Problem + +When Chromium renders an SVG to PDF via `page.pdf()` (Playwright) or `--print-to-pdf` (headless), `@font-face` rules placed only inside the SVG's `` block produce **Type 3 (bitmap) fonts** in the PDF: + +```xml + + + + + + Hello + +``` + +**Symptom:** PDF opens fine in browsers but shows rasterised/pixelated text in InDesign, Acrobat, or print-preflight; font is listed as Type 3 (not TrueType/CIDFont). + +## Root Cause + +Chromium's PDF renderer uses the HTML document's font-loading pipeline, not the SVG inline stylesheet, to decide which fonts to embed as vector outlines. `@font-face` inside SVG `` is applied for on-screen rendering but is **not** picked up by the PDF font-embedding step. + +## Fix + +Inject the same `@font-face` declaration into the **HTML document's ``** before calling `page.pdf()`: + +```python +# Playwright example +await page.set_content(f""" + + + + + + + {svg_content} + + +""") +await page.pdf(path="output.pdf", print_background=True) +``` + +The SVG `` stylesheet can remain (no need to remove it); the HTML `` declaration is additive and triggers proper CIDFont embedding. + +## Verification + +Open the resulting PDF in Acrobat → File → Properties → Fonts tab. The font should appear as **Type: TrueType (CID)** or **Type: OpenType (CID)**, not Type 3. + +## Scope + +Applies to: +- Playwright `page.pdf()` +- Puppeteer `page.pdf()` +- Chrome headless `--print-to-pdf` +- Any Chromium-based PDF renderer + +Does **not** apply to wkhtmltopdf (uses QtWebKit) or WeasyPrint (uses Cairo). + +## See Also + +- `wiki/concepts/chromium-pdf-css-font-injection` — this article diff --git a/wiki/concepts/python-float-formatting-comparison.md b/wiki/concepts/python-float-formatting-comparison.md new file mode 100644 index 0000000..155d635 --- /dev/null +++ b/wiki/concepts/python-float-formatting-comparison.md @@ -0,0 +1,64 @@ +--- +title: "Python Float Formatting — Equality Trap with Percentages" +tags: [python, float, formatting, gotcha] +created: 2026-05-18 +sources: daily/2026-05-18.md +--- + +# Python Float Formatting — Equality Trap with Percentages + +## The Problem + +Binary floating-point arithmetic breaks integer-equality checks on values that look whole: + +```python +1.1 * 10 # → 11.000000000000002 +(1.1 * 10) % 1 # → 2e-15 (not 0.0!) +(1.1 * 10) % 1 == 0 # → False ← BUG +``` + +A common context where this bites: formatting percentage strings. + +```python +def fmt_pct(value): + """Format 1.1 → '1.1%', 2.0 → '2%'""" + pct = value * 100 + if pct % 1 == 0: # WRONG — floats lie here + return f"{int(pct)}%" # never reached for 1.1 + return f"{pct:.2f}%" # outputs "1.10%" instead of "1.1%" +``` + +**Symptom:** Values like `1.1`, `3.3`, `6.6` produce `"1.10%"`, `"3.30%"`, `"6.60%"` instead of `"1.1%"`, `"3.3%"`, `"6.6%"`. + +## Root Cause + +IEEE 754 double precision cannot represent most decimal fractions exactly. Multiplying by 100 amplifies the error. The `% 1 == 0` check compares against exact zero — which the fractional residue never is. + +## Fix + +Format to a fixed number of decimal places first, then strip trailing zeros: + +```python +def fmt_pct(value): + pct = value * 100 + formatted = f"{pct:.10f}".rstrip("0").rstrip(".") + return f"{formatted}%" +``` + +Or use `Decimal` for exact arithmetic when the source is user-entered text: + +```python +from decimal import Decimal +def fmt_pct(value: str) -> str: + pct = Decimal(value) * 100 + return f"{pct.normalize():f}%" +``` + +## General Rule + +> Never use `float == integer` or `float % 1 == 0` for "is this whole?" checks. +> Always format to fixed dp first, then apply string operations. + +## See Also + +- `wiki/concepts/python-iso-z-suffix` — another float/string representation trap diff --git a/wiki/log.md b/wiki/log.md index 71935c1..ce2cf77 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -1,6 +1,16 @@ # Build Log +## [2026-05-18T21:30:00+01:00] compile | daily/2026-05-18.md — pass 3 (session 18:59 pimco-charts extraction) +- Source: daily/2026-05-18.md (session 18:59 — pimco-charts PDF rendering + chart generation) +- Articles created (2): + - [[wiki/concepts/python-float-formatting-comparison]] — `1.1 * 100 ≠ 110` in binary float; `% 1 == 0` returns False; fix: format to fixed dp then `rstrip("0")` + - [[wiki/concepts/chromium-pdf-css-font-injection]] — `@font-face` in SVG `` only → Type 3 bitmap fonts in PDF; must inject same CSS into HTML `` to get CIDFont (TrueType) embedding +- Mistakes appended (1): + - [[wiki/mistakes/general]] — 2026-05-18 float-percent-formatting: `"1.10%"` instead of `"1.1%"` due to float arithmetic; fix: rstrip pattern +- Index updates: concepts/_index.md (202→204, +2 entries), _master-index.md (concepts 202→204) +- Note: gemini.md mistake and gemini-preview-vs-stable-rate-limits concept were already compiled in pass 2 (13:18 session); this pass covers the second session only + ## [2026-05-18T21:00:00+01:00] compile | daily/2026-05-18.md — pass 2 (session 13:18 extraction) - Source: daily/2026-05-18.md (session 13:18 — BAIC Gemini 504/429 debugging) - Articles created (1): diff --git a/wiki/mistakes/general.md b/wiki/mistakes/general.md index 79768a5..c36c7e6 100644 --- a/wiki/mistakes/general.md +++ b/wiki/mistakes/general.md @@ -11,3 +11,18 @@ Running list. Newest first. Auto-populated from session flush. --- + +## 2026-05-18 — float-percent-formatting + +**Mistake:** Output `"1.10%"` instead of `"1.1%"` for percentage values like `1.1`. + +**Symptom:** Formatted percentage strings showed unnecessary trailing zeros (e.g. `"3.30%"`, `"6.60%"`) even for values that are exact to one decimal place. + +**Root cause:** `1.1 * 100 = 110.00000000000001` in binary float; `% 1 == 0` returns `False`; the "whole number" branch was never taken, falling through to `:.2f` formatting. + +**Fix:** Format to fixed dp first, then `rstrip("0").rstrip(".")`: +```python +f"{value * 100:.10f}".rstrip("0").rstrip(".") +``` + +**See also:** [[wiki/concepts/python-float-formatting-comparison]]