197 lines
No EOL
9.3 KiB
Markdown
197 lines
No EOL
9.3 KiB
Markdown
---
|
||
name: "DevOps ↔ ClickUp Sync"
|
||
client: Oliver Internal
|
||
status: active
|
||
tech: [Python, FastAPI, SQLAlchemy, SQLite, Docker, Pydantic, HTTPX]
|
||
local_path: /Users/ai_leed/Documents/Projects/Oliver/DevOps_Click_UP_sync
|
||
deploy: docker compose up -d --build
|
||
url: http://localhost:8080
|
||
tags: [oliver, devops, ado, clickup, sync, webhook]
|
||
created: 2026-04-14
|
||
server: local
|
||
port: 8080
|
||
db: SQLite
|
||
---
|
||
|
||
## Overview
|
||
|
||
DevOps_Click_UP_sync is a FastAPI webhook service that bi-directionally synchronises Azure DevOps work items with ClickUp tasks in real time. It creates, updates, and mirrors comments and attachments across both platforms, preventing echo-event loops through deduplication logic. The service exposes a simple web dashboard for managing project mappings and viewing sync history, and is deployable as a Docker container with SQLite persistence.
|
||
|
||
## Tech Stack
|
||
|
||
- **Frontend:** Vanilla JavaScript + HTML + Tailwind CSS (dashboard at `frontend/index.html`)
|
||
- **Backend:** Python 3.x + FastAPI + Uvicorn + Pydantic
|
||
- **Database:** SQLite with SQLAlchemy ORM (async via `aiosqlite`)
|
||
- **Infrastructure:** Docker Compose (single service, volume-mounted data directory)
|
||
- **AI/ML:** N/A
|
||
- **Key libraries:** `httpx` (async HTTP client for ADO/ClickUp REST APIs), `markdownify` (HTML↔Markdown conversion), `python-dotenv` (env config), `aiofiles` (async file I/O)
|
||
|
||
## Architecture
|
||
|
||
The service is a synchronisation engine that listens for webhooks from both Azure DevOps and ClickUp, then mirrors changes bidirectionally:
|
||
|
||
1. **Webhook Handlers** (`src/api/webhooks.py`):
|
||
- `POST /webhooks/ado` — receives ADO work item events
|
||
- `POST /webhooks/clickup` — receives ClickUp task events
|
||
- HMAC validation via `WEBHOOK_SECRET` prevents spoofing
|
||
|
||
2. **Sync Engine** (`src/sync/engine.py`):
|
||
- Core logic that determines what to create, update, or skip
|
||
- Calls Field Mapper to translate between ADO and ClickUp schemas
|
||
- Invokes REST clients to push changes back to origin system
|
||
|
||
3. **Field Mapper** (`src/sync/mapper.py`):
|
||
- Transforms ADO work item fields ↔ ClickUp task fields
|
||
- Handles title, description, state, assignee, priority, custom fields
|
||
- Converts HTML ↔ Markdown for comment bodies
|
||
|
||
4. **Echo-Event Deduplication** (`src/sync/dedup.py`):
|
||
- Maintains a `dedup` table: tracks (entity_id, direction, timestamp)
|
||
- Incoming webhook is **skipped** if the same entity was synced from the opposite direction within a TTL window (prevents A→B→A loops)
|
||
|
||
5. **REST Clients**:
|
||
- `src/clients/ado.py` — Azure DevOps REST API (work items, comments, attachments)
|
||
- `src/clients/clickup.py` — ClickUp API (tasks, comments, attachments)
|
||
|
||
6. **Database Models** (`src/database.py`):
|
||
- `SyncMap` — maps ADO work item ID ↔ ClickUp task ID
|
||
- `ProjectMap` — maps ADO project ↔ ClickUp space/folder
|
||
- `CommentMap` — maps ADO comment ↔ ClickUp comment for updates
|
||
- `DedupLog` — echo-event prevention records
|
||
|
||
7. **Dashboard APIs** (`src/api/dashboard.py`):
|
||
- `GET /api/dashboard/stats` — sync counts and health
|
||
- `GET /api/dashboard/project-maps` — view configured mappings
|
||
- `GET /api/dashboard/sync-log` — paginated sync history
|
||
|
||
8. **Setup API** (`src/api/setup.py`):
|
||
- `POST /api/setup/register-webhooks` — auto-registers webhook subscriptions with ADO/ClickUp
|
||
|
||
```
|
||
┌─────────────┐ ┌──────────────┐
|
||
│ Azure │ │ ClickUp │
|
||
│ DevOps │ │ │
|
||
└──────┬──────┘ └──────┬───────┘
|
||
│ │
|
||
│ POST /webhooks/ado │ POST /webhooks/clickup
|
||
│ (work item created/updated) │ (task created/updated)
|
||
│ │
|
||
└────────────┬─────────────────────────┘
|
||
│
|
||
┌───────▼────────┐
|
||
│ FastAPI │
|
||
│ Service │
|
||
│ (port 8080) │
|
||
└───────┬────────┘
|
||
│
|
||
┌────────────┼────────────┐
|
||
│ │ │
|
||
│ Sync │ Dedup │
|
||
│ Engine │ Check │
|
||
│ │ │
|
||
└──┬─────────┴────────┬───┘
|
||
│ │
|
||
┌─────▼──────┐ ┌──────▼────┐
|
||
│ Field │ │ Database │
|
||
│ Mapper │ │ (SQLite) │
|
||
│ │ │ │
|
||
└─────┬──────┘ └──────┬────┘
|
||
│ │
|
||
┌─────▼──────────────────▼────┐
|
||
│ REST Clients │
|
||
│ (ADO + ClickUp APIs) │
|
||
└──────────────────────────────┘
|
||
```
|
||
|
||
## Dev Commands
|
||
|
||
```bash
|
||
# Copy and configure environment variables
|
||
cp .env.example .env
|
||
# → Edit .env with ADO_PAT, CLICKUP_API_TOKEN, WEBHOOK_SECRET, PUBLIC_URL, etc.
|
||
|
||
# Start service in Docker (builds image, mounts ./data volume, runs on port 8080)
|
||
docker compose up -d
|
||
|
||
# Rebuild after code changes
|
||
docker compose up -d --build
|
||
|
||
# View live logs
|
||
docker compose logs -f sync-service
|
||
|
||
# Health check
|
||
curl http://localhost:8080/api/health
|
||
|
||
# Run locally without Docker (requires venv + pip install -r requirements.txt)
|
||
pip install -r requirements.txt
|
||
uvicorn src.main:app --host 0.0.0.0 --port 8080 --reload
|
||
|
||
# Access dashboard in browser
|
||
http://localhost:8080/
|
||
```
|
||
|
||
## Deployment
|
||
|
||
- **Server:** Unknown (no server hostname provided in config)
|
||
- **Deploy:** `docker compose up -d --build` (from project directory)
|
||
- **URL:** Controlled by `PUBLIC_URL` environment variable (must be HTTPS and reachable by ADO and ClickUp for webhook callbacks)
|
||
- **Port:** 8080
|
||
- **Service:** Docker Compose service named `sync-service` (not a systemd service)
|
||
- **Local path:** `/Users/ai_leed/Documents/Projects/Oliver/DevOps_Click_UP_sync`
|
||
- **Data persistence:** SQLite database stored in `./data/sync.db` (Docker volume `./data:/app/data`)
|
||
- **Health check:** Built-in Docker healthcheck (`GET /api/health` every 30s)
|
||
|
||
## Environment Variables
|
||
|
||
- `ADO_ORGANIZATION` — Azure DevOps organization name (required)
|
||
- `ADO_PROJECT` — Azure DevOps project name (required)
|
||
- `ADO_PAT` — Azure DevOps Personal Access Token with work item read/write scopes (required)
|
||
- `ADO_ASSIGNED_TO` — WIQL filter for assigned-to field; leave empty to sync all items, use `@Me` to sync all (optional)
|
||
- `CLICKUP_API_TOKEN` — ClickUp API token (format: `pk_...`); required
|
||
- `CLICKUP_WORKSPACE_ID` — ClickUp workspace ID (required)
|
||
- `WEBHOOK_SECRET` — Random secret for HMAC signature validation of incoming webhooks (required; **must not be `changeme`** or echo-event dedup will not activate)
|
||
- `PUBLIC_URL` — Public HTTPS URL where this service is reachable (required for webhook registration; e.g., `https://your-domain.com`)
|
||
- `LOG_LEVEL` — Logging level for uvicorn/app (`DEBUG`, `INFO`, `WARNING`, `ERROR`; default: `INFO`)
|
||
- `DATABASE_URL` — SQLAlchemy database URL (default: `sqlite+aiosqlite:///./data/sync.db`); supports PostgreSQL or MySQL if changed
|
||
|
||
## API / Endpoints
|
||
|
||
### Webhooks (External)
|
||
- `POST /webhooks/ado` — Azure DevOps webhook receiver (validates HMAC with `WEBHOOK_SECRET`)
|
||
- `POST /webhooks/clickup` — ClickUp webhook receiver (validates HMAC with `WEBHOOK_SECRET`)
|
||
|
||
### Setup
|
||
- `POST /api/setup/register-webhooks` — Automatically registers webhook subscriptions with ADO and ClickUp (idempotent)
|
||
|
||
### Dashboard
|
||
- `GET /` — Serve `frontend/index.html` (vanilla JS dashboard)
|
||
- `GET /api/health` — Health check endpoint (used by Docker healthcheck)
|
||
- `GET /api/dashboard/stats` — Sync statistics (total synced, last sync time, error count)
|
||
- `GET /api/dashboard/project-maps` — List all configured ADO↔ClickUp project mappings
|
||
- `GET /api/dashboard/sync-log?page=1&limit=20` — Paginated sync history log
|
||
|
||
### Config
|
||
- Routes defined in `src/main.py` (routers from `webhooks.py`, `setup.py`, `dashboard.py`)
|
||
|
||
## Known Issues
|
||
|
||
- **No test suite** — no unit or integration tests configured; no linting setup
|
||
- **Production hardening needed** — before deploying to production:
|
||
- Add pytest + test coverage
|
||
- Add black/flake8 linting
|
||
- Validate all exception handling (especially in sync engine)
|
||
- Add request rate-limiting
|
||
- Implement robust logging/monitoring
|
||
- **SSH access constraint** — no SSH to server without explicit user instruction
|
||
- **Field mapping gaps** — some custom fields may not be handled; ADO→ClickUp HTML→Markdown conversion may lose formatting in edge cases
|
||
- **No automatic webhook re-registration
|
||
|
||
## Sessions
|
||
### 2026-04-14 – Project catalogued
|
||
**Done:** Added to Obsidian second brain.
|
||
|
||
---
|
||
## Change Log
|
||
| Date | Requested | Changed | Files |
|
||
|------|-----------|---------|-------|
|
||
| 2026-04-14 | Initial setup | Note created | — | |