---
name: vault-agent
status: design
type: tool
tags:
  - type/project
  - domain/ai
  - tech/python
  - tech/ollama
created: 2026-05-18
---

# vault-agent — Design Spec

## Goal

A background daemon that maintains the Obsidian vault as a **living, transferable knowledge base**. Anyone (human or AI) who opens the vault should immediately understand its structure, navigate by links, and continue working without onboarding.

The agent runs outside Claude Code sessions — it is always on, not session-dependent.

---

## Success Criteria

- Every note in safe zones has complete frontmatter (name, status, tags, type, created)
- No isolated nodes — every note links to at least 2 others via `[[wikilinks]]`
- PARA structure is consistent — notes sit in the correct folder
- Vault graph goes from fragmented clusters → one connected body
- A new AI (Claude, GPT, local model) can open the vault and orient itself in < 3 hops

---

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                    vault-agent                       │
│                                                      │
│  ┌──────────┐    ┌────────────┐    ┌─────────────┐  │
│  │ Watcher  │    │  Startup   │    │    Idle     │  │
│  │(FSEvents)│    │  Batch     │    │  Processor  │  │
│  └────┬─────┘    └─────┬──────┘    └──────┬──────┘  │
│       └────────────────┴──────────────────┘         │
│                        │                            │
│                   ┌────▼─────┐                      │
│                   │ Queue    │  state.json           │
│                   └────┬─────┘  (resume on restart) │
│                        │                            │
│              ┌─────────▼──────────┐                 │
│              │    Processor       │                 │
│              │  1. exclusion check│                 │
│              │  2. embed (Qwen3)  │                 │
│              │  3. vector search  │                 │
│              │  4. build context  │                 │
│              │  5. Scout → JSON   │                 │
│              │  6. safe write     │                 │
│              │  7. update index   │                 │
│              └─────────┬──────────┘                 │
│                        │                            │
│          ┌─────────────┴──────────────┐             │
│          │         ChromaDB           │             │
│          │  (Qwen3-Embed, persistent) │             │
│          └────────────────────────────┘             │
└─────────────────────────────────────────────────────┘
```

---

## Three Triggers

| Trigger | When | What | Throttle |
|---------|------|------|---------|
| **Watcher** | File saved (FSEvents) | Process changed note | 3s debounce per file |
| **Startup batch** | macOS login (LaunchAgent) | Process all notes modified since `last_run` | 1 note / 1s |
| **Idle processor** | CPU < 20% sustained for 30s | Continue backlog queue | 1 note / 2s |

> Night shutdown is handled by `state.json` — agent resumes exactly where it stopped.

### state.json structure
```json
{
  "last_run": "2026-05-18T09:12:00",
  "files": {
    "02 Areas/Personal Goals.md": {
      "hash": "sha256:abc123",
      "processed_at": "2026-05-18T09:10:00",
      "version": 1
    }
  }
}
```

---

## Per-Note Processing Pipeline

```
1. EXCLUSION CHECK
   → skip if path matches exclusion list

2. PARSE FRONTMATTER
   → identify missing required fields

3. EMBED
   → Qwen3-Embed:0.6b via Ollama /api/embeddings

4. VECTOR SEARCH
   → ChromaDB cosine similarity, top-20 (threshold 0.65)
   → Scout's 10M context window lets us pass more context than typical 7B models

5. BUILD PROMPT CONTEXT
   → note content
   → top-20 similar note titles + first 200 chars each
   → PARA zone descriptions
   → tag taxonomy (from CLAUDE.md)
   → required frontmatter schema

6. LLAMA 4 SCOUT → structured JSON output
   {
     "frontmatter": { ... },      // only missing fields
     "links": ["[[note-a]]", ...], // confident links only
     "para_zone": "03 Resources", // suggested folder (if misplaced)
     "confidence": 0.87
   }

7. SAFE WRITE
   → frontmatter: merge additive (never overwrite existing values)
     EXCEPTION: `modified` field is always updated to today's date when agent writes
   → links: append to ## Related section (create if missing)
   → folder move: only if note has no frontmatter at all AND confidence > 0.85
     (higher threshold than links — moves are harder to reverse)
   → 2s iCloud sync safety delay before write

8. UPDATE INDEX
   → upsert embedding in ChromaDB
   → update state.json hash for this file
```

---

## Models

| Role | Model | Size | Notes |
|------|-------|------|-------|
| LLM | `llama4:scout` | ~23 GB (Q4_K_M) | 10M ctx, structured JSON output, MoE fast inference |
| Embeddings | `qwen3-embed:0.6b` | 370 MB | SOTA MTEB, sufficient for semantic search |

Both served via local Ollama. No internet required after initial pull.

---

## Exclusion List (hardcoded, never modified at runtime)

```python
EXCLUDED_DIRS = [
    "wiki/",
    "99 Daily/",
    "raw/",
    "Templates/",
    "output/",
    "Hoarder/",
    ".obsidian/",
    ".claude/",
    ".agents/",
    ".smart-env/",
    ".qwen/",
    "posts/",
    # 01 Projects/ is NOT fully excluded — see ALLOWED_IN_PROJECTS below
]

# In 01 Projects/, agent may ONLY write these specific filenames:
ALLOWED_IN_PROJECTS = ["USER_MANUAL.md", "DEVELOPER_MANUAL.md"]
# All other files in 01 Projects/ are read-only (for local_path extraction)

EXCLUDED_FILES = [
    "02 Areas/Pending Commands.md",
    "02 Areas/Work Tasks.md",
    "CLAUDE.md",
    "Next-Session-Prompt.md",
]
```

**Safe zones (processed):**
- `02 Areas/` (except excluded files above)
- `03 Resources/`
- `05 Aimpress LTD/`
- Root-level `.md` files (except excluded)

---

## Frontmatter Schema (from CLAUDE.md)

Required fields the agent enforces:

```yaml
name: "Human-readable title"
status: active | paused | archived | reference
type: project | area | resource | reference | sop | meeting
tags:
  - type/...
  - domain/...         # domain/ai, domain/devops, domain/analytics
  - client/...         # if applicable
  - tech/...           # if applicable
created: YYYY-MM-DD
modified: YYYY-MM-DD   # agent updates this on every write
```

Agent adds only missing fields. Never removes or overwrites.

---

## Linking Strategy

The `## Related` section is the **only place** the agent writes links into a note body.

```markdown
## Related

- [[note-a]] — brief reason why it's related
- [[wiki/concepts/some-concept]] — reference to wiki (read-only, outbound link only)
- [[03 Resources/SOP-name]]
```

If `## Related` already exists → agent appends new links only (no duplicates).

Minimum confidence for link insertion: **0.80**.

---

## PARA Classification

Used only for notes without existing frontmatter (first-time processing):

| Zone | Criteria |
|------|----------|
| `02 Areas/` | Ongoing responsibility, no end date (processes, personal systems) |
| `03 Resources/` | Reference material, topic of interest, cheatsheets, SOPs |
| `04 Archive/` | Agent never moves here — human decision only |
| `05 Aimpress LTD/` | Aimpress business matters |

Agent suggests folder in frontmatter (`suggested_folder` field) — auto-moves only if:
- Note has **no frontmatter at all**
- Confidence > 0.85
- File is in vault root (not already in a PARA folder)

---

## File Layout

```
~/.vault-agent/
├── vault_agent/
│   ├── agent.py           # entry point: starts watcher + idle processor + scheduler
│   ├── watcher.py         # FSEvents watchdog (watchdog lib)
│   ├── startup.py         # login batch + catch-up queue
│   ├── processor.py       # per-note enrichment pipeline (steps 1–8)
│   ├── indexer.py         # Qwen3-Embed + ChromaDB ops
│   ├── frontmatter.py     # safe YAML parse / merge / write
│   ├── classifier.py      # PARA zone + tag taxonomy
│   ├── context_builder.py # build Scout prompt from similar notes
│   ├── cleanup.py         # junk detection → Vault Cleanup.md report
│   ├── manuals.py         # codebase reader → USER_MANUAL + DEVELOPER_MANUAL
│   ├── ollama_client.py   # Ollama REST API (embed + generate)
│   └── config.py          # vault path, exclusions, model names, thresholds
├── db/                    # ChromaDB persistent storage
├── state.json             # {last_run, files: {path: {hash, processed_at, version}}}
├── vault-agent.log        # rotating, max 10 MB
├── pyproject.toml         # uv-managed deps
└── plist/
    └── com.vadym.vault-agent.plist  # LaunchAgent (login autostart)
```

---

## Dependencies

```toml
[project]
requires-python = ">=3.11"
dependencies = [
    "watchdog",           # FSEvents
    "chromadb",           # local vector store
    "python-frontmatter", # YAML parse/write
    "httpx",              # Ollama API calls
    "pydantic",           # JSON output validation from Scout
]
```

Managed with `uv`. No Poetry, no conda.

---

## LaunchAgent (macOS autostart)

`~/Library/LaunchAgents/com.vadym.vault-agent.plist`

- Starts at login
- Restarts automatically if crashed
- Ollama must be running (separate LaunchAgent or start-on-demand via `ollama serve`)
- Logs stdout/stderr to `~/.vault-agent/vault-agent.log`

---

## Safety Rules (non-negotiable)

1. **Read exclusion list before every write** — no exceptions
2. **Frontmatter merge is additive only** — existing values are sacred
3. **Links go only in `## Related`** — never in note body
4. **iCloud sync delay** — 2s after any file write before next write to same file
5. **No wiki/ writes** — agent can add outbound links TO wiki articles, but never edits wiki files
6. **No 04 Archive/ moves** — archiving is always a human decision
7. **state.json is the source of truth** — if file hash unchanged since last run, skip

---

## Initial Run (first launch)

1. Pull models: `ollama pull llama4:scout && ollama pull qwen3-embed:0.6b`
2. Index full vault: embed all non-excluded `.md` files → ChromaDB (~15–20 min for 1236 notes)
3. Build backlog queue: all notes missing frontmatter or `## Related`
4. Process queue at 1 note/2s with idle detection

Expected first-run time: **20–40 min** (background, no user action needed).

---

## Vault Cleanup

The agent identifies junk and surfaces it — **never deletes automatically**.

### What counts as junk

| Type | Detection |
|------|-----------|
| Empty note | Content < 50 chars after frontmatter strip |
| Stub note | Only frontmatter, no body |
| Broken [[link]] | Target file does not exist in vault |
| Duplicate title | Two notes with identical `name:` frontmatter field |
| MD5-named file at root | Files like `1960ec7d...MD5.png` cluttering root |

### What the agent does

1. Weekly scan (on startup if last cleanup scan > 7 days ago)
2. Collects junk candidates → writes `02 Areas/Vault Cleanup.md`
3. Never moves or deletes automatically — human reviews the list
4. User can add `cleanup: skip` to frontmatter to whitelist a note permanently

`02 Areas/Vault Cleanup.md` format:
```markdown
## Vault Cleanup — 2026-05-18

### Empty / Stub Notes (3)
- [[02 Areas/Old Idea.md]] — 12 chars, no content
...

### Broken Links (7)
- [[03 Resources/Guide.md]] references [[nonexistent-note]]
...

### Suggested Archive (2)
- [[02 Areas/2024 Project.md]] — status: archived, still in Areas/
...
```

---

## Project Manuals (Auto-Generated)

For each project in `01 Projects/`, the agent creates and maintains two documents:

### Target files (inside project folder, NOT conflicting with main project note)

```
01 Projects/{project}/
├── {Project Name}.md        ← Claude's domain (NOT touched)
├── USER_MANUAL.md           ← agent creates/updates
└── DEVELOPER_MANUAL.md      ← agent creates/updates
```

### Trigger

Manual update fires when:
- The project's codebase (`local_path` from frontmatter) has git commits newer than `manual_updated_at` in state.json
- Or manual is missing entirely

Agent reads `local_path` from the project note's frontmatter, then scans the codebase.

### USER_MANUAL.md — what it covers

Generated by reading: README, frontend routes/pages, key API endpoints.

```markdown
# {Project Name} — User Manual

## What This Tool Does
## Who Uses It (roles)
## How to Access
## Main Workflows (step-by-step)
## FAQ
```

### DEVELOPER_MANUAL.md — what it covers

Generated by reading: docker-compose, backend routes, models, key services, .env.example.

```markdown
# {Project Name} — Developer Manual

## Architecture Overview
## Tech Stack
## Local Setup
## Environment Variables
## Key Services & Entry Points
## API Reference (auto-extracted endpoints)
## Deployment
## Known Gotchas
```

### Safety rule for manuals

- Agent reads source code (read-only) to generate content
- Writes ONLY to `USER_MANUAL.md` and `DEVELOPER_MANUAL.md`
- Never touches the main project `.md` note
- Manuals get `auto_generated: true` + `manual_updated_at: YYYY-MM-DD` in frontmatter
- If human edits a manual → agent appends only, does not overwrite human sections
  (detects human edits via content hash of sections)

### Model for manuals

`llama4:scout` — large context needed to read full codebase files simultaneously.

---

## Vault Health Dashboard (optional Phase 2)

A note at `02 Areas/Vault Health.md` auto-updated by the agent:

```markdown
## Vault Health — 2026-05-18

| Metric | Value |
|--------|-------|
| Total notes (safe zones) | 312 |
| Notes with complete frontmatter | 287 (92%) |
| Notes with ≥1 Related link | 241 (77%) |
| Orphaned notes | 71 |
| Last full scan | 2026-05-18 09:12 |
```

---

## What This Does NOT Replace

| Claude Code task | Stays with Claude Code |
|-----------------|----------------------|
| Wiki compilation (`compile.py`) | Claude-managed pipeline |
| Session logging (Stop hook) | Claude Code hooks |
| Project note management (`01 Projects/`) | Claude during sessions |
| Complex reasoning / architecture decisions | Claude sessions |

The agent handles **mechanical vault hygiene**. Claude handles **knowledge synthesis**.
