semblance-dev/README.md

# Semblance Synthetic Society

An AI-powered platform for creating synthetic personas and running autonomous focus group sessions. Build realistic consumer profiles, moderate live discussions or let AI drive the conversation, and extract actionable themes and insights — all in real time.

## Features

### Persona Management
- **AI Persona Generation** — Generate detailed synthetic personas from audience briefs using multi-model LLM support
- **Manual Creation** — Build custom personas with demographic, psychographic, and behavioral attributes
- **Folder Organization** — Organize personas into folders with drag-and-drop support
- **Bulk Export** — Export persona profiles individually or in bulk (PDF, summary formats)
- **Persona Modification** — Refine and adjust AI-generated personas after creation

### Focus Group Sessions
- **Manual Moderation** — Guide discussions in real time as a human moderator
- **Autonomous AI Mode** — Let the AI runner orchestrate multi-persona conversations with intelligent speaker selection and conversation flow decisions
- **Real-Time WebSocket Communication** — Live message streaming via Socket.IO with room-based session management
- **Discussion Guide Generation** — AI-generated structured discussion guides from research objectives
- **Theme Extraction** — Automatic identification and highlighting of key themes across session transcripts
- **Session Analytics** — Summary generation and insight extraction from completed sessions
- **AI Moderator** — AI-assisted moderation with probe generation and follow-up questions

### AI Integration
- **Multi-Model Support** — Google Gemini 3 Pro (default), OpenAI GPT-4.1, OpenAI GPT-5.2
- **Prompt Template System** — 20 markdown-based prompt templates for persona generation, conversation management, theme extraction, and more
- **Configurable Reasoning** — Adjustable reasoning effort for supported models

### Authentication
- **Local Login** — Username/password authentication with JWT tokens
- **Microsoft 365 SSO** — Azure AD integration via MSAL for enterprise environments

### Dashboard & Organization
- **Project Dashboard** — Overview of personas, focus groups, and recent activity
- **Folder System** — Hierarchical organization with drag-and-drop (DND Kit)
- **Charts & Analytics** — Visual insights with Recharts

## Tech Stack

### Frontend
- **Build**: Vite 5, TypeScript 5.5
- **UI**: React 18, Tailwind CSS, shadcn-ui (Radix UI), Lucide React icons
- **State**: TanStack Query, React Hook Form + Zod validation
- **Routing**: React Router DOM
- **Real-Time**: Socket.IO client
- **Charts**: Recharts
- **Drag & Drop**: DND Kit
- **Auth**: MSAL React (Azure AD)

### Backend
- **Framework**: Quart (async Python), Hypercorn ASGI server
- **Database**: MongoDB via PyMongo + Motor (async)
- **Real-Time**: python-socketio (AsyncServer)
- **AI/LLM**: OpenAI SDK, Google Generative AI (`google-genai`)
- **Auth**: Custom Quart-compatible JWT, MSAL (Microsoft), bcrypt

### Infrastructure
- **Database**: MongoDB
- **Deployment**: systemd service, automated `deploy.sh` script
- **Process**: Hypercorn ASGI (port 5137)

## Getting Started

### Prerequisites

- **Node.js** (v18+) & npm — [install with nvm](https://github.com/nvm-sh/nvm#installing-and-updating)
- **Python 3.11+**
- **MongoDB** installed and running (default: `mongodb://localhost:27017`)

### Installation

```sh
# Clone the repository
git clone https://github.com/your-org/synthetic-society.git
cd synthetic-society

# Install frontend dependencies
npm install

# Set up the backend
cd backend
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cd ..
```

### Environment Setup

Copy the appropriate environment file:

```sh
# For local development
cp .env.development .env

# For production
cp .env.production .env
```

See [Environment Configuration](#environment-configuration) for details on each setting.

### Running the Application

**Option 1 — Start script** (starts both frontend and backend):

```sh
./start.sh
```

**Option 2 — Run separately:**

```sh
# Terminal 1: Start the backend
cd backend
source venv/bin/activate
python run.py

# Terminal 2: Start the frontend
npm run dev
```

- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:5137/api

### Default Login

- **Username**: `user`
- **Password**: `pass`

> Local login is enabled by default in development. In production, Microsoft SSO is the primary auth method.

## Project Structure

```
├── src/                          # Frontend source
│   ├── components/
│   │   ├── ui/                   # shadcn-ui components
│   │   ├── focus-group-session/  # Session UI (DiscussionPanel, ParticipantPanel, ThemesPanel)
│   │   ├── persona/              # Persona management components
│   │   ├── ai-recruiter/         # AI persona recruitment
│   │   ├── auth/                 # Authentication components
│   │   └── dashboard/            # Dashboard components
│   ├── pages/                    # Route pages (Dashboard, FocusGroups, Login, etc.)
│   ├── hooks/                    # Custom hooks (WebSocket, personas, discussion guides)
│   ├── contexts/                 # React contexts (Auth, WebSocket, Navigation)
│   ├── services/                 # WebSocket service layer
│   ├── lib/                      # API client, utilities
│   ├── types/                    # TypeScript type definitions
│   ├── utils/                    # Avatar, mention, discussion guide utilities
│   └── config/                   # MSAL configuration
├── backend/
│   ├── run.py                    # Entry point (Hypercorn ASGI)
│   ├── app/
│   │   ├── routes/               # API endpoints (auth, personas, focus-groups, ai-personas, folders, tasks)
│   │   ├── services/             # Business logic (19 services)
│   │   ├── models/               # Data models (User, Persona, FocusGroup, Folder)
│   │   ├── auth/                 # Custom Quart JWT implementation
│   │   └── utils/                # Prompt loader, discussion guide schema
│   └── prompts/                  # LLM prompt templates (20 markdown files)
├── deploy.sh                     # Production deployment script
├── semblance.service             # systemd service file
├── start.sh                      # Local development start script
├── .env.development              # Development environment config
└── .env.production               # Production environment config
```

## Architecture Overview

### Real-Time Communication
Socket.IO provides bidirectional WebSocket communication between the React frontend and Quart backend. The `websocket_manager_async.py` module manages room-based messaging so each focus group session operates in an isolated channel. Messages are broadcast to all participants in a room as they arrive.

### Autonomous Conversation System
Focus groups can run without human moderation:
- **AI Runner Service** (`ai_runner_service.py`) — Manages background task execution for autonomous sessions
- **Autonomous Conversation Controller** (`autonomous_conversation_controller.py`) — Orchestrates the multi-persona conversation loop
- **Conversation Decision Service** (`conversation_decision_service.py`) — Determines when to continue, who speaks next, and when to wrap up
- **Conversation Context Service** (`conversation_context_service.py`) — Maintains conversation state, history, and context window

### LLM Integration Layer
The `llm_service.py` module provides a unified interface across multiple AI providers:
- **Google Gemini** (`gemini-3-pro-preview`) — Default model
- **OpenAI** (`gpt-4.1`, `gpt-5.2`) — Alternative models with reasoning effort support
- Prompt templates in `/backend/prompts/` are loaded by `prompt_loader.py` and injected at runtime

## Environment Configuration

| Setting | Development | Production |
|---|---|---|
| **Base Path** | `/` | `/semblance/` |
| **API Base URL** | `/api` (proxied to `:5137`) | `https://optical-dev.oliver.solutions/semblance_back/api` |
| **WebSocket Path** | `/socket.io/` | `/semblance_back/socket.io/` |
| **MSAL Redirect** | `http://localhost:5173/` | `https://optical-dev.oliver.solutions/semblance` |
| **Local Login** | Enabled | Disabled |

Environment files:
- **`.env.development`** — Local development settings
- **`.env.production`** — Production server settings
- **`.env`** — Active config (copy from the appropriate file above)

## Deployment

### Build

```sh
npm run build          # Production frontend build
```

### Deploy

The project includes an automated deployment script:

```sh
./deploy.sh
```

This script handles:
1. Pulling latest code from git
2. Setting up the Python virtual environment
3. Installing backend dependencies
4. Building the frontend
5. Deploying built assets to the web server directory
6. Restarting the `semblance.service` systemd unit

### Manual Backend Start (Production)

```sh
cd backend
source venv/bin/activate
hypercorn "app:create_app()" --bind 0.0.0.0:5137
```

The included `semblance.service` file can be installed as a systemd unit for process management.

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request