# Oliver Metadata Tool - FastAPI Backend

Complete FastAPI backend migration from Flask with Redis sessions, JWT authentication, and full API.

## ✅ What's Complete

### Backend (100%)
- ✅ FastAPI app with async I/O
- ✅ Redis session storage (solves session loss problem!)
- ✅ JWT authentication (access + refresh tokens)
- ✅ Microsoft SSO support
- ✅ File upload/download with persistent storage
- ✅ All metadata sources: AI, Excel, Import, Manual, Templates
- ✅ All processors copied from Flask (100% working as-is)
- ✅ SQLAlchemy async database
- ✅ Docker Compose setup

### API Endpoints (17 total)
- Auth: `/auth/login`, `/auth/logout`, `/auth/token/refresh`, `/auth/register`
- Files: `/files/upload`, `/files/{file_id}/download`, `/files/download-batch`
- Metadata: `/metadata/{file_id}`, `/metadata/batch-update`
- Templates: `/templates/` (list, create, get, delete, preview)

## 🚀 Quick Start

### Option 1: Docker Compose (Recommended)

```bash
# 1. Copy environment file
cp .env.fastapi.example .env

# 2. Edit .env and add your OpenAI API key
nano .env

# 3. Start services
docker-compose -f docker-compose.fastapi.yml up -d

# 4. Check logs
docker-compose -f docker-compose.fastapi.yml logs -f backend

# 5. Access API
open http://localhost:8000/docs
```

### Option 2: Local Development

```bash
# 1. Install Redis
brew install redis  # macOS
# or: sudo apt-get install redis-server  # Linux

# 2. Start Redis
redis-server

# 3. Create virtual environment
cd backend
python3 -m venv venv
source venv/bin/activate

# 4. Install dependencies
pip install -r requirements.txt

# 5. Copy environment file
cp ../.env.fastapi.example ../.env

# 6. Edit .env
nano ../.env

# 7. Run backend
python -m app.main

# 8. Access API
open http://localhost:8000/docs
```

## 📝 Configuration

### Required Environment Variables

```env
# OpenAI API key (required for AI metadata generation)
OPENAI_API_KEY=sk-...

# Secret key for JWT tokens (generate new one!)
SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")

# Redis URL
REDIS_URL=redis://localhost:6379/0
```

### Optional Environment Variables

```env
# Database (default: SQLite)
DATABASE_URL=sqlite+aiosqlite:///./data/oliver_metadata.db

# Microsoft SSO
AZURE_CLIENT_ID=...
AZURE_CLIENT_SECRET=...
AZURE_TENANT_ID=...

# Frontend URL for CORS
FRONTEND_URL=http://localhost:3000
```

## 🧪 Testing the API

### 1. Create a Test User

```bash
curl -X POST http://localhost:8000/auth/register \
  -H "Content-Type: application/json" \
  -d '{"username": "testuser", "password": "testpass"}'
```

### 2. Login and Get Tokens

```bash
curl -X POST http://localhost:8000/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "testuser", "password": "testpass"}'
```

Response:
```json
{
  "access_token": "eyJ...",
  "refresh_token": "eyJ...",
  "token_type": "bearer",
  "expires_in": 1800,
  "user": {...}
}
```

### 3. Upload Files

```bash
# Save access token
TOKEN="your-access-token-here"

# Upload file with AI metadata
curl -X POST http://localhost:8000/files/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "files=@test.pdf" \
  -F "metadata_source=ai"
```

### 4. Update Metadata

```bash
curl -X PUT http://localhost:8000/metadata/FILE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "SESSION_ID",
    "file_index": 0,
    "metadata": {
      "title": "Updated Title",
      "subject": "Updated Subject",
      "keywords": "test, metadata"
    }
  }'
```

### 5. Download File

```bash
curl -X GET http://localhost:8000/files/FILE_ID/download \
  -H "Authorization: Bearer $TOKEN" \
  --output downloaded_file.pdf
```

## 📚 Interactive API Documentation

FastAPI provides automatic interactive API docs:

- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc

You can test all endpoints directly in the browser!

## 🔧 Architecture

### Session Management (CRITICAL FIX)

**Before (Flask):**
- In-memory dict: `sessions = {}`
- Lost on restart ❌

**After (FastAPI):**
- Redis with TTL
- Persistent across restarts ✅
- User sessions: 7 days
- File sessions: 1 hour
- Auto-cleanup

### Authentication Flow

1. Login → JWT access token (30 min) + refresh token (7 days)
2. Refresh token stored in Redis
3. Frontend sends: `Authorization: Bearer <access_token>`
4. Token expired? → Use refresh token to get new access token
5. Logout → Delete session from Redis

### File Processing Flow

1. Upload files → Save to `uploads/{user_id}/{YYYYMMDD}/`
2. Create session in Redis with file info
3. Generate metadata (AI/Excel/Import/Manual/Template)
4. User reviews/edits metadata
5. Update file with metadata
6. Download processed file
7. Cleanup (automatic after 7 days)

## 🐳 Docker Services

### Running Services

```bash
# Start all services
docker-compose -f docker-compose.fastapi.yml up -d

# View logs
docker-compose -f docker-compose.fastapi.yml logs -f

# Stop services
docker-compose -f docker-compose.fastapi.yml down

# Rebuild backend
docker-compose -f docker-compose.fastapi.yml build backend
docker-compose -f docker-compose.fastapi.yml up -d backend
```

### Service URLs

- **Backend API**: http://localhost:8000
- **API Docs**: http://localhost:8000/docs
- **Redis**: localhost:6379
- **PostgreSQL**: localhost:5432 (optional)

## 🗄️ Database

### SQLite (Default)

Location: `backend/data/oliver_metadata.db`

**Pros:**
- Simple, no setup
- Good for single server
- Easy migration from Flask

**Cons:**
- No concurrent writes
- Not for multi-server deployment

### PostgreSQL (Optional)

**Pros:**
- Better performance
- Concurrent connections
- Multi-server support

**To enable:**

```yaml
# docker-compose.fastapi.yml
environment:
  DATABASE_URL: postgresql+asyncpg://oliver:${DB_PASSWORD}@postgres:5432/oliver_metadata
```

## 📦 What's Reused from Flask

These components are **100% unchanged**:

- `backend/app/processors/extractors/` - All file extractors
- `backend/app/processors/updaters/` - All file updaters
- `backend/app/processors/metadata_analyzer.py` - AI generation
- `backend/app/processors/excel_metadata_lookup.py` - Excel lookup
- `backend/app/processors/template_manager.py` - Templates
- `backend/app/processors/config.py` - Configuration

**Zero modifications needed** - they work perfectly with FastAPI!

## 🔒 Security

### Production Checklist

- [ ] Change `SECRET_KEY` to random 64-char string
- [ ] Enable HTTPS (set `REDIRECT_URI` to https://)
- [ ] Restrict CORS origins in `main.py`
- [ ] Set `DEBUG=false` in production
- [ ] Use PostgreSQL instead of SQLite for multi-server
- [ ] Enable Redis password: `redis://user:password@host:6379/0`
- [ ] Regular backups of database and uploads
- [ ] Monitor Redis memory usage

## 🐛 Troubleshooting

### Redis Connection Error

```bash
# Check if Redis is running
redis-cli ping
# Should return: PONG

# If not running:
redis-server
```

### Database Lock Error

```bash
# SQLite only - check if another process is using DB
lsof backend/data/oliver_metadata.db

# If stuck, delete and restart:
rm backend/data/oliver_metadata.db
docker-compose -f docker-compose.fastapi.yml restart backend
```

### Import Errors

```bash
# Check if all dependencies installed
cd backend
pip list | grep fastapi
pip list | grep redis

# If missing:
pip install -r requirements.txt
```

### File Upload 413 Error

```bash
# Increase max file size in main.py or nginx.conf
# Default: 500MB (configured in processors/config.py)
```

## 📈 Monitoring

### Check Redis Sessions

```bash
# Connect to Redis
redis-cli

# List all session keys
KEYS *session*

# Get session data
GET file_session:SESSION_ID

# Check memory usage
INFO memory
```

### Check Storage

```bash
# Get storage stats
curl http://localhost:8000/files/stats \
  -H "Authorization: Bearer $TOKEN"
```

### Check Logs

```bash
# Docker logs
docker-compose -f docker-compose.fastapi.yml logs -f backend

# Or if running locally
# Logs printed to console
```

## 🚧 What's Next (Frontend)

To complete the migration:

1. Create React frontend (see plan in `.claude/plans/`)
2. Implement file upload UI with drag-drop
3. Metadata editor components
4. Template management UI
5. Import/Excel mapping modals

Backend is **100% ready** for frontend integration!

## 📞 Support

- **API Documentation**: http://localhost:8000/docs
- **Migration Plan**: `.claude/plans/radiant-snacking-chipmunk.md`
- **Memory**: `.claude/projects/.../memory/MEMORY.md`

---

**Status**: ✅ Backend Complete | ⏳ Frontend Pending

Generated with Claude Code by Anthropic