vault backup: 2026-05-18 18:29:59

01 Projects/DevOps_Click_UP_sync/DEVELOPER_MANUAL.md

@@ -0,0 +1,124 @@
+---
+auto_generated: true
+created: 2026-05-18
+manual_updated_at: 2026-05-18
+modified: 2026-05-18
+name: Developer_Manual
+status: active
+tags:
+- domain/devops
+- tech/azure-ad
+- tech/docker
+- tech/fastapi
+- tech/python
+- tech/sqlite
+- type/sop
+type: sop
+---
+
+# DevOps ↔ ClickUp Sync Developer Manual
+
+## Architecture Overview
+The application is a FastAPI-based microservice that listens for webhooks from Azure DevOps and ClickUp. It processes these events, determines the direction of sync, and updates the respective API. A SQLite database persists the mapping between entity IDs to prevent duplicate creation and ensure referential integrity.
+
+## Tech Stack
+*   **Framework:** FastAPI (Python 3.10+)
+*   **Database:** SQLite (via `aiosqlite` for async support)
+*   **ORM:** SQLAlchemy (Async)
+*   **HTTP Client:** `httpx`
+*   **Configuration:** Pydantic Settings
+*   **Deployment:** Docker / Docker Compose
+
+## Local Setup
+
+### 1. Prerequisites
+*   Python 3.10+
+*   Docker & Docker Compose
+
+### 2. Installation
+```bash
+# Clone the repo
+git clone <repo-url>
+cd devops-clickup-sync
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Copy env file
+cp .env.example .env
+
+# Edit .env with your credentials
+```
+
+### 3. Run Locally
+```bash
+# Using Python directly
+uvicorn src.main:app --reload --host 0.0.0.0 --port 8080
+
+# Or using Docker
+docker-compose up -d
+```
+
+## Environment Variables
+See `.env.example` for all required variables.
+
+*   `ADO_PAT`: Personal Access Token for Azure DevOps (Scope: `Work Item (Read & Write)`, `Code (Read)`).
+*   `CLICKUP_API_TOKEN`: Private App token from ClickUp (Scope: `Write` for tasks/comments).
+*   `WEBHOOK_SECRET`: Used to validate incoming webhooks.
+*   `DATABASE_URL`: Defaults to local SQLite. Change to PostgreSQL/MySQL URI for production.
+
+## Key Services & Entry Points
+
+### `src/main.py`
+*   **Entry Point:** Initializes FastAPI app, lifespan events (DB init), and mounts routers.
+*   **Routers:**
+    *   `webhook_router`: Handles incoming POST requests from ADO/ClickUp.
+    *   `dashboard_router`: Serves `/api/health`, `/api/status`.
+    *   `setup_router`: Manages mapping configurations.
+
+### `src/database.py`
+*   **Models:**
+    *   `SyncMap`: Links `ado_work_item_id` to `clickup_task_id`.
+    *   `ProjectMap`: Maps ADO Projects/Areas to ClickUp Spaces/Lists.
+    *   `CommentMap`: Links `ado_comment_id` to `clickup_comment_id`.
+    *   `SyncLog`: Audit trail for sync events.
+*   **Init:** `init_db()` creates tables if they don't exist.
+
+### `src/clients/ado.py`
+*   `ADOClient`: Wraps Azure DevOps REST API.
+*   Key methods: `get_work_item`, `update_work_item`, `create_work_item`.
+
+### `src/clients/clickup.py`
+*   `ClickUpClient`: Wraps ClickUp API v2.
+*   Key methods: `get_task`, `update_task`, `create_task`, `get_spaces/lists`.
+
+## API Reference
+
+### Webhooks
+*   **Method:** POST
+*   **URL:** `PUBLIC_URL/webhook` (or configured route)
+*   **Headers:** `X-Webhook-Secret` must match `WEBHOOK_SECRET`.
+*   **Body:** JSON Payload from ADO (`src/models/ado.py`) or ClickUp (`src/models/clickup.py`).
+
+### Dashboard API
+*   **GET** `/api/health`: Returns `{"status": "ok"}`.
+*   **GET** `/api/status`: Returns counts of synced items, comment maps, and recent sync log statuses.
+
+## Deployment
+
+### Docker Compose
+The `docker-compose.yml` defines the service:
+*   Port 8080 exposed.
+*   Data persisted in `./data` volume.
+*   Healthcheck configured on `/api/health`.
+
+### Production Considerations
+1.  **Database:** Switch `DATABASE_URL` to a managed PostgreSQL instance. Migrate SQLAlchemy models accordingly.
+2.  **Security:** Use a reverse proxy (Nginx/Traefik) for SSL termination. Do not expose port 8080 directly to the internet.
+3.  **Scaling:** This stateless service can be scaled horizontally, but ensure webhooks are distributed correctly and DB connections are pooled.
+
+## Known Gotchas
+1.  **Circular Webhooks:** Ensure you filter out updates caused by *your own* syncs to prevent infinite loops. Check `sync_source` in `SyncMap` before sending updates.
+2.  **ADO Area Paths:** Mapping is done via `ado_area_path`. Complex project structures require explicit mapping in `ProjectMap`.
+3.  **ClickUp Task Descriptions:** Long descriptions may be truncated. Ensure API limits are respected.
+4.  **Rate Limiting:** Both ADO and ClickUp have rate limits. Monitor `SyncLog` for 429 errors and implement backoff strategies if necessary.

01 Projects/DevOps_Click_UP_sync/USER_MANUAL.md

@@ -0,0 +1,90 @@
+---
+auto_generated: true
+created: 2026-05-18
+manual_updated_at: 2026-05-18
+modified: 2026-05-18
+name: User_Manual
+status: active
+tags:
+- domain/devops
+- tech/azure-ad
+- tech/docker
+- tech/fastapi
+- type/sop
+type: sop
+---
+
+# DevOps ↔ ClickUp Sync User Manual
+
+## What This Tool Does
+The **DevOps ↔ ClickUp Sync** service is an automated bridge between Microsoft Azure DevOps (ADO) and ClickUp. It synchronizes work items (tasks, bugs, epics) and their associated comments between the two platforms in real-time.
+
+**Key Capabilities:**
+*   **Bidirectional Sync:** Updates made in ADO are reflected in ClickUp, and vice versa.
+*   **Comment Sync:** Comments added in either platform are pushed to the other.
+*   **Mapping Management:** Automatically links corresponding IDs between ADO Work Items and ClickUp Tasks.
+*   **Dashboard Visibility:** Provides a real-time status overview of sync operations.
+
+## Who Uses It
+*   **Project Managers:** Who need to track progress in ClickUp while developers work in Azure DevOps.
+*   **Developers:** Who need visibility into task statuses managed by their PMs in ClickUp.
+*   **DevOps Engineers:** Who automate the synchronization pipeline.
+
+## How to Access
+
+### 1. Web Dashboard
+Access the user interface via your configured public URL:
+*   **URL:** `https://your-domain.com` (or `http://localhost:8080` if running locally)
+*   **Features:** View sync status, recent logs, and connection health.
+
+### 2. API Health Check
+Verify the service is running via the API endpoint:
+*   **GET** `http://localhost:8080/api/health`
+*   **Expected Response:** `{"status": "ok", "timestamp": "..."}`
+
+## Main Workflows
+
+### Workflow 1: Initial Setup
+1.  **Configure Environment Variables:**
+    *   Set `ADO_ORGANIZATION`, `ADO_PROJECT`, and `ADO_PAT` (Personal Access Token with read/write permissions).
+    *   Set `CLICKUP_API_TOKEN` (from ClickUp User Settings > Apps > Private Apps).
+    *   Set `CLICKUP_WORKSPACE_ID`.
+2.  **Start the Service:**
+    *   Run `docker-compose up -d`.
+3.  **Verify Connection:**
+    *   Check the Dashboard UI for "Status: OK".
+    *   Check logs for successful database initialization.
+
+### Workflow 2: Synchronizing Tasks
+1.  **Create an Item in Source System:**
+    *   Create a Task in ClickUp or a Work Item in ADO.
+2.  **Trigger Sync:**
+    *   Ensure Webhooks are configured in both ADO (Notifications > New Subscription) and ClickUp (Task > Settings > Webhooks).
+    *   The payload sent to `PUBLIC_URL` triggers the sync logic.
+3.  **Verify Sync:**
+    *   Check the **Sync Log** in the Dashboard.
+    *   Look for the linked ID in the other system (e.g., "ADO ID: 123" in ClickUp description or custom fields).
+
+### Workflow 3: Synchronizing Comments
+1.  **Add Comment:**
+    *   Post a comment on an ADO Work Item or ClickUp Task.
+2.  **Automatic Propagation:**
+    *   The webhook triggers the `CommentMap` logic.
+    *   The comment appears in the other system.
+
+## FAQ
+
+**Q: How do I map an ADO Project to a ClickUp Space/List?**
+A: This is handled automatically if `CLICKUP_WORKSPACE_ID` is correct and the sync runs. You can also use the setup endpoint (`/api/setup`) to manually bind specific ADO Area Paths to ClickUp Lists.
+
+**Q: Which tasks are synced?**
+A: By default, only tasks assigned to the user associated with the `ADO_PAT` (indicated by `@Me`). Change `ADO_ASSIGNED_TO` in `.env` to sync all tasks or specific users.
+
+**Q: What if a sync fails?**
+A: Check the `SyncLog` table or Dashboard. Common causes include invalid webhooks, network timeouts, or permission issues with the PAT/Token.
+
+**Q: Can I undo a sync?**
+A: No, the sync is immediate. Always review changes in the Dashboard log before assuming they are permanent.
+
+## Related
+- [[01 Projects/DevOps_Click_UP_sync/DEVELOPER_MANUAL.md]]

01 Projects/ac-tool/DEVELOPER_MANUAL.md

@@ -0,0 +1,178 @@
+---
+auto_generated: true
+created: 2026-05-18
+manual_updated_at: 2026-05-18
+modified: 2026-05-18
+name: Developer_Manual
+status: active
+tags:
+- domain/devops
+- tech/azure-ad
+- tech/docker
+- tech/fastapi
+- tech/postgresql
+- tech/react
+- tech/typescript
+type: reference
+---
+
+# AC Tool Developer Manual
+
+## Architecture Overview
+
+AC Tool follows a three-tier architecture:
+1. **Frontend**: React + Vite + TailwindCSS, managed via Zustand for state.
+2. **Backend**: Python FastAPI (implied by docker-compose and typical patterns), serving REST API and WebSockets.
+3. **Database**: PostgreSQL 16, stored in a Docker volume.
+
+### Communication Flow
+- **HTTP**: REST API for CRUD operations (clients, sheets, jobs, auth).
+- **WebSocket**: Real-time job progress updates.
+- **Storage**: Local filesystem for uploaded files (`./data` in Docker), PostgreSQL for metadata.
+
+## Tech Stack
+
+- **Frontend**: React, TypeScript, Vite, TailwindCSS, Zustand, MSAL.js (implied by Azure config)
+- **Backend**: Python, FastAPI (inferred), Uvicorn (inferred), MSAL (Azure Auth), OpenAI/Gemini/Anthropic SDKs
+- **Database**: PostgreSQL 16
+- **Infrastructure**: Docker Compose, Apache Reverse Proxy (production), Vite Dev Server (local)
+
+## Local Setup
+
+### Prerequisites
+- Node.js 18+
+- Python 3.10+
+- Docker & Docker Compose
+- Git
+
+### Steps
+1. **Clone the repository**
+   ```bash
+   git clone <repo-url>
+   cd ac-tool
+   ```
+
+2. **Set up environment variables**
+   ```bash
+   cp .env.example .env
+   # Edit .env with your Azure AD, API keys, and DB password
+   ```
+
+3. **Start Backend & Database**
+   ```bash
+   docker-compose up -d postgres app
+   ```
+
+4. **Install Frontend Dependencies**
+   ```bash
+   cd frontend
+   npm install
+   ```
+
+5. **Run Frontend Dev Server**
+   ```bash
+   npm run dev
+   ```
+   The app will be available at `http://localhost:5173` and proxied to `http://localhost:8000` for API/WS.
+
+## Environment Variables
+
+### Critical Variables (.env)
+- `POSTGRES_PASSWORD`: Database password (default: `achelper_secret`)
+- `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_REDIRECT_URI`: For SSO
+- `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ANTHROPIC_API_KEY`: AI provider keys
+- `EMERGENCY_TOKEN`: For bypassing SSO (leave blank to disable)
+- `APP_PORT`: Host port for the backend (default: 8100)
+
+### Azure AD Configuration
+- The app uses MSAL with PKCE flow (no client secret).
+- Redirect URI must match `AZURE_REDIRECT_URI` in Azure Portal.
+
+## Key Services & Entry Points
+
+### Frontend Stores (Zustand)
+- `useAuthStore`: Handles authentication state and token management.
+- `useClientStore`: Manages client CRUD operations.
+- `useJobStore`: Handles job creation, deletion, and real-time updates.
+- `useSheetStore`: Manages spreadsheet data and deliverables.
+- `useDropdownStore`: Fetches and caches category/media type data.
+
+### Backend Entry Points (Inferred)
+- `/auth/me`: GET - Fetch current user info.
+- `/api/clients`: CRUD for clients.
+- `/api/sheets`: CRUD for sheets and deliverables.
+- `/api/jobs`: POST (create job), GET (list), WebSocket connection.
+- `/api/dropdowns`: GET categories for dropdowns.
+
+### AI Providers
+- **OpenAI**: Configured via `OPENAI_API_KEY`, model `gpt-4.1`.
+- **Google Gemini**: Configured via `GEMINI_API_KEY`, model `gemini-2.0-flash-exp`.
+- **Anthropic Claude**: Configured via `ANTHROPIC_API_KEY`, models `claude-opus-4-5-20251101` and `claude-sonnet-4-5-20250929`.
+
+## API Reference
+
+### Auth
+- `GET /auth/me`: Returns `User` object.
+
+### Clients
+- `GET /api/clients`: List all clients.
+- `POST /api/clients`: Create client (name).
+- `PUT /api/clients/{id}`: Update client.
+- `DELETE /api/clients/{id}`: Delete client.
+
+### Sheets
+- `GET /api/sheets`: List sheets.
+- `POST /api/sheets`: Create sheet (name).
+- `GET /api/sheets/{id}`: Load sheet data.
+- `PUT /api/sheets/{id}`: Save sheet data.
+- `DELETE /api/sheets/{id}`: Delete sheet.
+- `POST /api/sheets/{id}/duplicate`: Duplicate sheet.
+
+### Jobs
+- `POST /api/jobs`: Upload files and start job.
+- `GET /api/jobs`: List jobs.
+- `DELETE /api/jobs/{id}`: Cancel/delete job.
+- `GET /ws`: WebSocket endpoint for real-time job updates.
+
+### Dropdowns
+- `GET /api/dropdowns/{clientId?}`: Get categories and media types.
+
+## Deployment
+
+### Production Environment
+1. **Apache Configuration**: Use the provided Apache VirtualHost snippet in `docker-compose.yml` to proxy requests.
+2. **SSL**: Ensure SSL termination is handled by Apache or the load balancer.
+3. **Database**: Use a managed PostgreSQL service or ensure the Docker volume is backed up.
+4. **Environment Variables**: Set all required `.env` variables on the host.
+5. **Docker**: Run `docker-compose up -d` to start services.
+
+### Nginx Alternative
+If using Nginx instead of Apache, configure:
+```nginx
+location /ac-helper/api/ {
+    proxy_pass http://127.0.0.1:8100/api/;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+}
+location /ac-helper/ws {
+    proxy_pass http://127.0.0.1:8100/ws;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+}
+location /ac-helper/ {
+    alias /var/www/html/ac-helper/;
+    index index.html;
+}
+```
+
+## Known Gotchas
+
+1. **Azure AD Redirect URI**: Must exactly match the configured `AZURE_REDIRECT_URI` including trailing slashes.
+2. **Emergency Token**: Must be a long random string (use `secrets.token_hex(32)`). Leaving it blank disables emergency login.
+3. **WebSocket Path**: The frontend strips `/ac-helper` before proxying. Ensure backend WebSocket path is `/ws`.
+4. **CORS**: Vite dev server handles CORS via proxy. In production, rely on Apache/Nginx reverse proxy; do not use CORS headers from the backend.
+5. **Database Connection**: The backend uses `DATABASE_URL` with `postgres:5432` (internal Docker hostname), not `localhost`.
+6. **Model Costs**: AI processing costs accumulate quickly. Monitor `costUsdTotal` in job summaries and set `minimum_success_threshold` if needed.
+7. **File Uploads**: Large files may timeout if `OPENAI_TIMEOUT` or similar is too low. Default is 3600s.
+8. **State Hydration**: Frontend stores do not persist across refreshes except for `activeSheetId` in localStorage. Users may need to re-login or reload data.

01 Projects/ac-tool/USER_MANUAL.md

@@ -0,0 +1,100 @@
+---
+auto_generated: true
+created: 2026-05-18
+manual_updated_at: 2026-05-18
+modified: 2026-05-18
+name: User_Manual
+status: active
+tags:
+- domain/ai
+- tech/azure-ad
+- tech/fastapi
+- tech/react
+- type/sop
+type: sop
+---
+
+# AC Tool User Manual
+
+## What This Tool Does
+
+AC Tool is a comprehensive platform for managing creative deliverables, specifically designed for media production and advertising workflows. It automates the extraction of data from spreadsheets, analyzes content using AI models (OpenAI, Google Gemini, Anthropic), and consolidates results into structured CSV outputs. The tool supports multi-client environments, custom dropdowns, and real-time collaboration.
+
+### Key Features:
+- **AI-Powered Spreadsheet Analysis**: Upload Excel/CSV files to extract and classify deliverables automatically.
+- **Multi-Provider AI Integration**: Switch between OpenAI, Google Gemini, and Anthropic Claude models for analysis.
+- **Client Management**: Manage multiple clients with custom categories and media types.
+- **Real-Time Job Tracking**: Monitor AI processing jobs with live progress updates and cost tracking.
+- **Emergency Access**: Bypass SSO for emergency situations using token-based login.
+
+## Who Uses It
+
+- **Media Production Managers**: Track deliverables across multiple projects and formats.
+- **Advertising Agencies**: Manage client-specific dropdowns and categories.
+- **Content Teams**: Automate data extraction from large spreadsheets.
+- **Administrators**: Manage users, clients, and system configurations.
+
+## How to Access
+
+### Standard Login
+1. Navigate to `https://ai-sandbox.oliver.solutions/ac-helper/`
+2. Click "Sign In" in the top right corner.
+3. Authenticate via Azure AD (SSO) using your organizational credentials.
+
+### Emergency Access
+1. Navigate to the login page.
+2. Look for the "Emergency Access" link (visible only when EMERGENCY_TOKEN is configured).
+3. Enter the emergency token and user credentials provided by your administrator.
+
+### API Access
+- REST API endpoints are available at `/ac-helper/api/`
+- WebSocket updates at `/ac-helper/ws/`
+- Use the `ac_access_token` stored in sessionStorage for authentication.
+
+## Main Workflows
+
+### 1. Managing Clients
+1. Go to the "Clients" section.
+2. Click "Create Client" to add a new client.
+3. Enter the client name and save.
+4. Configure custom dropdowns and categories for the client (Admin only).
+
+### 2. Uploading and Processing a Spreadsheet
+1. Navigate to "Jobs" or the upload area.
+2. Click "Upload File" and select your Excel/CSV file.
+3. Choose the AI model configuration (optional).
+4. Click "Process" to start the job.
+5. Monitor progress in real-time via the job card.
+6. Download the resulting CSV once the job completes.
+
+### 3. Managing Sheets
+1. Go to "Sheets" to view all saved spreadsheets.
+2. Click on a sheet to load it into the editor.
+3. Edit deliverables manually if needed.
+4. Click "Save" to persist changes.
+5. Use "Duplicate" to create a copy of an existing sheet.
+
+### 4. Viewing Job History
+1. Navigate to the "Jobs" section.
+2. Filter by status, client, or date.
+3. Click on a job to view details: cost, tokens used, latency, and results.
+
+## FAQ
+
+**Q: How much does AI processing cost?**
+A: Each job displays its total cost in USD. You can configure minimum success thresholds to stop processing early if costs exceed limits.
+
+**Q: What AI models are supported?**
+A: Currently supports OpenAI (gpt-4.1), Google Gemini (gemini-2.0-flash-exp), and Anthropic Claude (Opus, Sonnet).
+
+**Q: Can I use the tool without Azure AD?**
+A: Yes, if your administrator has configured an EMERGENCY_TOKEN. Contact your admin for access.
+
+**Q: How do I handle large files?**
+A: The tool supports large uploads, but processing time and token usage will increase. Consider splitting large files into smaller batches for better performance.
+
+**Q: Is data stored securely?**
+A: Yes, all data is stored in a PostgreSQL database with encrypted connections. Access is controlled via Azure AD and role-based permissions.
+
+**Q: Can I export my data?**
+A: Yes, all sheets can be exported as CSV, and jobs include downloadable result files.