--- name: "DevOps ↔ ClickUp Sync" client: Oliver Internal status: active tech: [Python, FastAPI, SQLAlchemy, SQLite, Docker, Pydantic, HTTPX] local_path: /Users/ai_leed/Documents/Projects/Oliver/DevOps_Click_UP_sync deploy: docker compose up -d --build url: http://localhost:8080 tags: [oliver, devops, ado, clickup, sync, webhook] created: 2026-04-14 server: local port: 8080 db: SQLite --- ## Overview DevOps_Click_UP_sync is a FastAPI webhook service that bi-directionally synchronises Azure DevOps work items with ClickUp tasks in real time. It creates, updates, and mirrors comments and attachments across both platforms, preventing echo-event loops through deduplication logic. The service exposes a simple web dashboard for managing project mappings and viewing sync history, and is deployable as a Docker container with SQLite persistence. ## Tech Stack - **Frontend:** Vanilla JavaScript + HTML + Tailwind CSS (dashboard at `frontend/index.html`) - **Backend:** Python 3.x + FastAPI + Uvicorn + Pydantic - **Database:** SQLite with SQLAlchemy ORM (async via `aiosqlite`) - **Infrastructure:** Docker Compose (single service, volume-mounted data directory) - **AI/ML:** N/A - **Key libraries:** `httpx` (async HTTP client for ADO/ClickUp REST APIs), `markdownify` (HTML↔Markdown conversion), `python-dotenv` (env config), `aiofiles` (async file I/O) ## Architecture The service is a synchronisation engine that listens for webhooks from both Azure DevOps and ClickUp, then mirrors changes bidirectionally: 1. **Webhook Handlers** (`src/api/webhooks.py`): - `POST /webhooks/ado` — receives ADO work item events - `POST /webhooks/clickup` — receives ClickUp task events - HMAC validation via `WEBHOOK_SECRET` prevents spoofing 2. **Sync Engine** (`src/sync/engine.py`): - Core logic that determines what to create, update, or skip - Calls Field Mapper to translate between ADO and ClickUp schemas - Invokes REST clients to push changes back to origin system 3. **Field Mapper** (`src/sync/mapper.py`): - Transforms ADO work item fields ↔ ClickUp task fields - Handles title, description, state, assignee, priority, custom fields - Converts HTML ↔ Markdown for comment bodies 4. **Echo-Event Deduplication** (`src/sync/dedup.py`): - Maintains a `dedup` table: tracks (entity_id, direction, timestamp) - Incoming webhook is **skipped** if the same entity was synced from the opposite direction within a TTL window (prevents A→B→A loops) 5. **REST Clients**: - `src/clients/ado.py` — Azure DevOps REST API (work items, comments, attachments) - `src/clients/clickup.py` — ClickUp API (tasks, comments, attachments) 6. **Database Models** (`src/database.py`): - `SyncMap` — maps ADO work item ID ↔ ClickUp task ID - `ProjectMap` — maps ADO project ↔ ClickUp space/folder - `CommentMap` — maps ADO comment ↔ ClickUp comment for updates - `DedupLog` — echo-event prevention records 7. **Dashboard APIs** (`src/api/dashboard.py`): - `GET /api/dashboard/stats` — sync counts and health - `GET /api/dashboard/project-maps` — view configured mappings - `GET /api/dashboard/sync-log` — paginated sync history 8. **Setup API** (`src/api/setup.py`): - `POST /api/setup/register-webhooks` — auto-registers webhook subscriptions with ADO/ClickUp ``` ┌─────────────┐ ┌──────────────┐ │ Azure │ │ ClickUp │ │ DevOps │ │ │ └──────┬──────┘ └──────┬───────┘ │ │ │ POST /webhooks/ado │ POST /webhooks/clickup │ (work item created/updated) │ (task created/updated) │ │ └────────────┬─────────────────────────┘ │ ┌───────▼────────┐ │ FastAPI │ │ Service │ │ (port 8080) │ └───────┬────────┘ │ ┌────────────┼────────────┐ │ │ │ │ Sync │ Dedup │ │ Engine │ Check │ │ │ │ └──┬─────────┴────────┬───┘ │ │ ┌─────▼──────┐ ┌──────▼────┐ │ Field │ │ Database │ │ Mapper │ │ (SQLite) │ │ │ │ │ └─────┬──────┘ └──────┬────┘ │ │ ┌─────▼──────────────────▼────┐ │ REST Clients │ │ (ADO + ClickUp APIs) │ └──────────────────────────────┘ ``` ## Dev Commands ```bash # Copy and configure environment variables cp .env.example .env # → Edit .env with ADO_PAT, CLICKUP_API_TOKEN, WEBHOOK_SECRET, PUBLIC_URL, etc. # Start service in Docker (builds image, mounts ./data volume, runs on port 8080) docker compose up -d # Rebuild after code changes docker compose up -d --build # View live logs docker compose logs -f sync-service # Health check curl http://localhost:8080/api/health # Run locally without Docker (requires venv + pip install -r requirements.txt) pip install -r requirements.txt uvicorn src.main:app --host 0.0.0.0 --port 8080 --reload # Access dashboard in browser http://localhost:8080/ ``` ## Deployment - **Server:** Unknown (no server hostname provided in config) - **Deploy:** `docker compose up -d --build` (from project directory) - **URL:** Controlled by `PUBLIC_URL` environment variable (must be HTTPS and reachable by ADO and ClickUp for webhook callbacks) - **Port:** 8080 - **Service:** Docker Compose service named `sync-service` (not a systemd service) - **Local path:** `/Users/ai_leed/Documents/Projects/Oliver/DevOps_Click_UP_sync` - **Data persistence:** SQLite database stored in `./data/sync.db` (Docker volume `./data:/app/data`) - **Health check:** Built-in Docker healthcheck (`GET /api/health` every 30s) ## Environment Variables - `ADO_ORGANIZATION` — Azure DevOps organization name (required) - `ADO_PROJECT` — Azure DevOps project name (required) - `ADO_PAT` — Azure DevOps Personal Access Token with work item read/write scopes (required) - `ADO_ASSIGNED_TO` — WIQL filter for assigned-to field; leave empty to sync all items, use `@Me` to sync all (optional) - `CLICKUP_API_TOKEN` — ClickUp API token (format: `pk_...`); required - `CLICKUP_WORKSPACE_ID` — ClickUp workspace ID (required) - `WEBHOOK_SECRET` — Random secret for HMAC signature validation of incoming webhooks (required; **must not be `changeme`** or echo-event dedup will not activate) - `PUBLIC_URL` — Public HTTPS URL where this service is reachable (required for webhook registration; e.g., `https://your-domain.com`) - `LOG_LEVEL` — Logging level for uvicorn/app (`DEBUG`, `INFO`, `WARNING`, `ERROR`; default: `INFO`) - `DATABASE_URL` — SQLAlchemy database URL (default: `sqlite+aiosqlite:///./data/sync.db`); supports PostgreSQL or MySQL if changed ## API / Endpoints ### Webhooks (External) - `POST /webhooks/ado` — Azure DevOps webhook receiver (validates HMAC with `WEBHOOK_SECRET`) - `POST /webhooks/clickup` — ClickUp webhook receiver (validates HMAC with `WEBHOOK_SECRET`) ### Setup - `POST /api/setup/register-webhooks` — Automatically registers webhook subscriptions with ADO and ClickUp (idempotent) ### Dashboard - `GET /` — Serve `frontend/index.html` (vanilla JS dashboard) - `GET /api/health` — Health check endpoint (used by Docker healthcheck) - `GET /api/dashboard/stats` — Sync statistics (total synced, last sync time, error count) - `GET /api/dashboard/project-maps` — List all configured ADO↔ClickUp project mappings - `GET /api/dashboard/sync-log?page=1&limit=20` — Paginated sync history log ### Config - Routes defined in `src/main.py` (routers from `webhooks.py`, `setup.py`, `dashboard.py`) ## Known Issues - **No test suite** — no unit or integration tests configured; no linting setup - **Production hardening needed** — before deploying to production: - Add pytest + test coverage - Add black/flake8 linting - Validate all exception handling (especially in sync engine) - Add request rate-limiting - Implement robust logging/monitoring - **SSH access constraint** — no SSH to server without explicit user instruction - **Field mapping gaps** — some custom fields may not be handled; ADO→ClickUp HTML→Markdown conversion may lose formatting in edge cases - **No automatic webhook re-registration ## Sessions ### 2026-04-14 – Project catalogued **Done:** Added to Obsidian second brain. --- ## Change Log | Date | Requested | Changed | Files | |------|-----------|---------|-------| | 2026-04-14 | Initial setup | Note created | — |