cohorta/backend
Vadym Samoilenko d0ad8e67be Fix backfill: use accumulated conversation context for prompt estimation
Old logic used output text length as a proxy for prompt tokens — completely
wrong. Real Gemini calls send the full conversation history as context, so
prompt grows with every turn.

New logic:
- completion_tokens = len(response_text) / 3.8 (what was generated)
- prompt_tokens = base_template + sum(all_prior_messages_in_fg) / 3.8
  - persona_response base: 1500 tok (template + persona details + topic)
  - moderator base: 1200 tok (moderator template + fg context)
  - persona_generate base: 2500 tok (persona-detailed-generation.md template)

Also:
- Sorts messages chronologically per focus group before processing
- Accumulates context correctly so turn N includes turns 0..N-1 as context
- Idempotency via pre-fetched set instead of per-doc find_one queries
- cost_usd breakdown now has correct input/output split (not 40/60 guess)
- Dry-run prints per-focus-group cost estimates for sanity checking

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 19:11:01 +01:00
..
app Add period selector to all cost-bearing admin tabs 2026-04-24 19:03:16 +01:00
prompts changed permissions 2025-12-19 19:26:16 +00:00
scripts Fix backfill: use accumulated conversation context for prompt estimation 2026-04-24 19:11:01 +01:00
tests Add LLM usage tracking infrastructure (Phases A-C) 2026-04-24 18:08:27 +01:00
uploads/focus-group-68af42ff19ed40daa02b0392 changed permissions 2025-12-19 19:26:16 +00:00
.dockerignore Fix .dockerignore — exclude *.txt but keep requirements.txt 2026-03-20 13:28:32 +00:00
.DS_Store changed permissions 2025-12-19 19:26:16 +00:00
.env.example Apply Jintech security audit remediation (sprint 3) — 87/92 findings fixed 2026-03-20 12:51:18 +00:00
Dockerfile Dockerize backend — replace systemd service with docker-compose 2026-03-20 13:21:34 +00:00
hypercorn_config.py changed permissions 2025-12-19 19:26:16 +00:00
logging_config.py changed permissions 2025-12-19 19:26:16 +00:00
migrate_legacy_folders.py truncated long folder names so the buttons are still visible, migrated legacy folders to new format 2025-09-11 10:40:01 -05:00
MIGRATION_README.md changed permissions 2025-12-19 19:26:16 +00:00
README.md changed permissions 2025-12-19 19:26:16 +00:00
requirements.txt Add LLM usage tracking infrastructure (Phases A-C) 2026-04-24 18:08:27 +01:00
run.py Remove create_default_user call from run.py (method removed in security remediation) 2026-03-20 13:34:26 +00:00
test_asset.txt changed permissions 2025-12-19 19:26:16 +00:00
test_image.png changed permissions 2025-12-19 19:26:16 +00:00
test_websocket_cross_process.py changed permissions 2025-12-19 19:26:16 +00:00

Semblance Synthetic Society Backend

This is the Python backend for the Semblance Synthetic Society project. It provides API endpoints for authentication, personas, and focus groups.

Setup

  1. Make sure you have Python 3.8+ installed
  2. Create a virtual environment:
    cd backend
    python -m venv venv
    
  3. Activate the virtual environment:
    • On macOS/Linux:
      source venv/bin/activate
      
    • On Windows:
      venv\Scripts\activate
      
  4. Install dependencies:
    pip install -r requirements.txt
    

Running the Backend

python run.py

The server will start on http://localhost:5000

API Endpoints

Authentication

  • POST /api/auth/login - Login with username and password
  • POST /api/auth/register - Register a new user
  • GET /api/auth/me - Get current user profile

Personas

  • GET /api/personas - Get personas for current user
  • GET /api/personas/all - Get all personas
  • GET /api/personas/:id - Get persona by ID
  • POST /api/personas - Create a new persona
  • PUT /api/personas/:id - Update a persona
  • DELETE /api/personas/:id - Delete a persona
  • POST /api/personas/batch - Create multiple personas

Focus Groups

  • GET /api/focus-groups - Get focus groups for current user
  • GET /api/focus-groups/all - Get all focus groups
  • GET /api/focus-groups/:id - Get focus group by ID
  • POST /api/focus-groups - Create a new focus group
  • PUT /api/focus-groups/:id - Update a focus group
  • DELETE /api/focus-groups/:id - Delete a focus group
  • POST /api/focus-groups/:id/participants - Add participant to focus group
  • DELETE /api/focus-groups/:id/participants/:personaId - Remove participant from focus group
  • GET /api/focus-groups/:id/messages - Get messages for a focus group
  • POST /api/focus-groups/:id/messages - Add a message to a focus group

AI Personas

  • POST /api/ai-personas/generate - Generate a synthetic persona using AI
  • POST /api/ai-personas/generate-and-save - Generate and save a synthetic persona
  • POST /api/ai-personas/batch-generate - Generate multiple synthetic personas
  • POST /api/ai-personas/batch-generate-and-save - Generate and save multiple synthetic personas

Focus Group AI

  • POST /api/focus-group-ai/generate-response - Generate an AI response from a persona in a focus group discussion

AI Response Generation Example

Request Body:

{
  "focus_group_id": "focus_group_id",
  "persona_id": "persona_id",
  "current_topic": "What do you think about this product?",
  "temperature": 0.7  // Optional, controls randomness (0.0 to 1.0)
}

Response:

{
  "message": "Response generated successfully",
  "response": "I find the product quite interesting. As someone who values efficiency, I appreciate the intuitive interface and how it streamlines my workflow. However, I'm concerned about the price point, which seems high compared to similar options on the market.",
  "message_id": "message_id"
}

How AI Response Generation Works

The system generates realistic persona responses by:

  1. Using the persona's demographic details, personality traits, goals, and frustrations
  2. Including the full discussion guide text
  3. Taking up to 50 most recent conversation messages for context
  4. Processing the current topic/question
  5. Generating a response in the persona's authentic voice

The current_topic parameter can be any text: a moderator question, a specific prompt, or a summary of discussion points. The AI will respond as if the persona is directly addressing this topic.

Default User

A default user with the following credentials is automatically created:

  • Username: user
  • Password: pass
  • Role: admin