initial commit

.claude/settings.local.json

@@ -0,0 +1,23 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(chmod:*)",
+      "Bash(grep:*)",
+      "Bash(ls:*)",
+      "Bash(mkdir:*)",
+      "Bash(npm run lint)",
+      "Bash(npx eslint:*)",
+      "Bash(npm run build:dev:*)",
+      "WebFetch(domain:ai-sandbox.oliver.solutions)",
+      "Bash(curl:*)",
+      "Bash(rg:*)",
+      "Bash(sed:*)",
+      "Bash(npm run build:*)",
+      "Bash(source:*)",
+      "Bash(python:*)",
+      "Bash(find:*)"
+    ],
+    "deny": []
+  },
+  "enableAllProjectMcpServers": false
+}

.env

@@ -0,0 +1,5 @@
+# Frontend URL
+VITE_FRONTEND_BASE_URL=https://ai-sandbox.oliver.solutions/semblance
+
+# Backend API URL
+VITE_API_BASE_URL=https://ai-sandbox.oliver.solutions/semblance_back/api

CLAUDE.md

@@ -0,0 +1,29 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+- Build: `npm run build` (use this for all testing and verification)
+- Lint code: `npm run lint`
+- Preview production build: `npm run preview`
+- Python testing: When modifying Python files, activate the virtual environment with `source backend/venv/bin/activate` and test syntax by importing the module with `python -c "import app.services.module_name"`
+
+**Note**: This project is hosted on a server. Always use `npm run build` instead of `npm run dev` for testing changes.
+
+**Python Testing**: After modifying any Python files in the backend, ALWAYS activate the virtual environment and test for syntax errors by attempting to import the modified module. This catches syntax errors before deployment.
+
+## Code Style Guidelines
+- **Imports**: Group imports by source (React, third-party, local)
+- **Types**: Use TypeScript. Project allows nullable types (`strictNullChecks: false`) 
+- **Components**: Use functional components with hooks
+- **Naming**: Use PascalCase for components, camelCase for variables/functions
+- **Formatting**: Follow ESLint recommendations, focus on readability
+- **Error Handling**: Use try/catch blocks with toast for user feedback
+- **CSS**: Use Tailwind classes for styling, with component-specific CSS files when needed
+- **File Structure**: Components in `/src/components`, pages in `/src/pages`, hooks in `/src/hooks`
+- **UI Components**: Use shadcn-ui components from `/src/components/ui`
+- **State Management**: React hooks for local state, context/props for sharing
+- **URL Construction**: ALWAYS use `import.meta.env.BASE_URL` when constructing URLs for static assets, images, or links. This project uses base path `/semblance/` in production. Example: `${import.meta.env.BASE_URL}image.png` instead of `/image.png`
+
+## Project Stack
+Vite, React, TypeScript, Tailwind CSS, shadcn-ui

README.md

@@ -0,0 +1,134 @@
+# Semblance Synthetic Society
+
+A platform for creating and managing synthetic personas for focus groups and market research.
+
+## Project info
+
+**URL**: https://lovable.dev/projects/ee7a424f-7f6c-4b5d-9645-e66074cea7d3
+
+## Features
+
+- Create and manage synthetic personas with detailed profiles
+- Organize personas into focus groups
+- Run interactive focus group sessions
+- Analyze results and extract insights
+- MongoDB-based backend for data persistence
+- User authentication and access control
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js & npm installed - [install with nvm](https://github.com/nvm-sh/nvm#installing-and-updating)
+- Python 3.8+ installed for the backend
+- MongoDB installed and running locally (default configuration: mongodb://localhost:27017)
+
+### Installation
+
+```sh
+# Step 1: Clone the repository
+git clone <YOUR_GIT_URL>
+
+# Step 2: Navigate to the project directory
+cd <YOUR_PROJECT_NAME>
+
+# Step 3: Install frontend dependencies
+npm install
+
+# Step 4: Install backend dependencies
+cd backend
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+pip install -r requirements.txt
+cd ..
+```
+
+### Running the Application
+
+Use the provided start script to run both frontend and backend:
+
+```sh
+./start.sh
+```
+
+The start script will:
+1. Check for and start MongoDB if needed
+2. Set up the Python virtual environment
+3. Install dependencies
+4. Populate the database with sample personas and focus groups
+5. Start both the backend and frontend servers
+
+Or run them separately:
+
+```sh
+# Start the backend
+cd backend
+source venv/bin/activate
+python run.py
+
+# In another terminal, run the frontend
+npm run dev
+```
+
+The frontend will be available at http://localhost:5173
+The backend API is available at http://localhost:5137/api
+
+### Default Login
+
+- Username: user
+- Password: pass
+
+## Technology Stack
+
+### Frontend
+- Vite
+- TypeScript
+- React
+- React Router
+- shadcn-ui
+- Tailwind CSS
+- Axios for API requests
+
+### Backend
+- Python
+- Flask
+- PyMongo (MongoDB client)
+- JWT for authentication
+
+## Project Structure
+
+- `/src`: Frontend source code
+  - `/components`: React components
+  - `/contexts`: React contexts for state management
+  - `/hooks`: Custom React hooks
+  - `/lib`: Utility functions and API client
+  - `/pages`: Main application pages
+  - `/types`: TypeScript type definitions
+- `/backend`: Python backend
+  - `/app`: Flask application
+    - `/models`: Database models
+    - `/routes`: API endpoints
+  - `run.py`: Backend entry point
+
+## Deployment
+
+The application is configured to be deployed at the `/semblance/` path. For hosting:
+
+1. Build the frontend:
+   ```sh
+   npm run build
+   ```
+
+2. Deploy the backend using a WSGI server like Gunicorn:
+   ```sh
+   cd backend
+   gunicorn -w 4 "app:create_app()"
+   ```
+
+## Contributing
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request

backend/.env

@@ -0,0 +1,12 @@
+# MongoDB Configuration - these are the MongoDB admin credentials, not app credentials
+MONGO_URI=mongodb://localhost:27017/semblance_db
+
+# If you need to connect to MongoDB with authentication, uncomment and set these values
+# MONGO_USER=admin
+# MONGO_PASSWORD=password
+
+# Flask app settings
+FLASK_APP=run.py
+FLASK_DEBUG=1
+# FLASK_ENV is deprecated in Flask 2.x, using FLASK_DEBUG instead
+SECRET_KEY=your-secret-key-for-sessions-and-tokens

backend/.env.example

@@ -0,0 +1,10 @@
+# Application Settings
+SECRET_KEY=your_secret_key_here
+JWT_SECRET_KEY=your_jwt_secret_key_here
+
+# MongoDB Settings
+MONGO_URI=mongodb://localhost:27017/
+
+# Environment
+FLASK_APP=run.py
+FLASK_ENV=development

backend/README.md

@@ -0,0 +1,115 @@
+# Semblance Synthetic Society Backend
+
+This is the Python backend for the Semblance Synthetic Society project. It provides API endpoints for authentication, personas, and focus groups.
+
+## Setup
+
+1. Make sure you have Python 3.8+ installed
+2. Create a virtual environment:
+   ```bash
+   cd backend
+   python -m venv venv
+   ```
+3. Activate the virtual environment:
+   - On macOS/Linux:
+     ```bash
+     source venv/bin/activate
+     ```
+   - On Windows:
+     ```cmd
+     venv\Scripts\activate
+     ```
+4. Install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+## Running the Backend
+
+```bash
+python run.py
+```
+
+The server will start on http://localhost:5000
+
+## API Endpoints
+
+### Authentication
+
+- `POST /api/auth/login` - Login with username and password
+- `POST /api/auth/register` - Register a new user
+- `GET /api/auth/me` - Get current user profile
+
+### Personas
+
+- `GET /api/personas` - Get personas for current user
+- `GET /api/personas/all` - Get all personas
+- `GET /api/personas/:id` - Get persona by ID
+- `POST /api/personas` - Create a new persona
+- `PUT /api/personas/:id` - Update a persona
+- `DELETE /api/personas/:id` - Delete a persona
+- `POST /api/personas/batch` - Create multiple personas
+
+### Focus Groups
+
+- `GET /api/focus-groups` - Get focus groups for current user
+- `GET /api/focus-groups/all` - Get all focus groups
+- `GET /api/focus-groups/:id` - Get focus group by ID
+- `POST /api/focus-groups` - Create a new focus group
+- `PUT /api/focus-groups/:id` - Update a focus group
+- `DELETE /api/focus-groups/:id` - Delete a focus group
+- `POST /api/focus-groups/:id/participants` - Add participant to focus group
+- `DELETE /api/focus-groups/:id/participants/:personaId` - Remove participant from focus group
+- `GET /api/focus-groups/:id/messages` - Get messages for a focus group
+- `POST /api/focus-groups/:id/messages` - Add a message to a focus group
+
+### AI Personas
+
+- `POST /api/ai-personas/generate` - Generate a synthetic persona using AI
+- `POST /api/ai-personas/generate-and-save` - Generate and save a synthetic persona
+- `POST /api/ai-personas/batch-generate` - Generate multiple synthetic personas
+- `POST /api/ai-personas/batch-generate-and-save` - Generate and save multiple synthetic personas
+
+### Focus Group AI
+
+- `POST /api/focus-group-ai/generate-response` - Generate an AI response from a persona in a focus group discussion
+
+#### AI Response Generation Example
+
+**Request Body**:
+```json
+{
+  "focus_group_id": "focus_group_id",
+  "persona_id": "persona_id",
+  "current_topic": "What do you think about this product?",
+  "temperature": 0.7  // Optional, controls randomness (0.0 to 1.0)
+}
+```
+
+**Response**:
+```json
+{
+  "message": "Response generated successfully",
+  "response": "I find the product quite interesting. As someone who values efficiency, I appreciate the intuitive interface and how it streamlines my workflow. However, I'm concerned about the price point, which seems high compared to similar options on the market.",
+  "message_id": "message_id"
+}
+```
+
+#### How AI Response Generation Works
+
+The system generates realistic persona responses by:
+
+1. Using the persona's demographic details, personality traits, goals, and frustrations
+2. Including the full discussion guide text
+3. Taking up to 50 most recent conversation messages for context
+4. Processing the current topic/question
+5. Generating a response in the persona's authentic voice
+
+The current_topic parameter can be any text: a moderator question, a specific prompt, or a summary of discussion points. The AI will respond as if the persona is directly addressing this topic.
+
+## Default User
+
+A default user with the following credentials is automatically created:
+- Username: user
+- Password: pass
+- Role: admin

backend/app/__init__.py

@@ -0,0 +1,106 @@
+from flask import Flask
+from flask_cors import CORS
+from flask_jwt_extended import JWTManager
+from dotenv import load_dotenv
+import os
+import tempfile
+
+load_dotenv()
+
+def setup_temp_directories():
+    """Set up temporary directories for Flask/Werkzeug file handling."""
+    # Try to create a temp directory in the backend folder
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    temp_dir = os.path.join(backend_dir, 'temp')
+    upload_dir = os.path.join(backend_dir, 'uploads')
+    
+    # Create directories with proper permissions
+    for directory in [temp_dir, upload_dir]:
+        try:
+            os.makedirs(directory, exist_ok=True)
+            os.chmod(directory, 0o755)
+            
+            # Test write permissions
+            test_file = os.path.join(directory, 'test_write')
+            with open(test_file, 'w') as f:
+                f.write('test')
+            os.remove(test_file)
+            print(f"✓ Directory {directory} is writable")
+            
+        except (OSError, PermissionError) as e:
+            print(f"Warning: Cannot write to {directory}: {e}")
+            continue
+    
+    # Set environment variables for Python's tempfile module
+    if os.path.isdir(temp_dir) and os.access(temp_dir, os.W_OK):
+        os.environ['TMPDIR'] = temp_dir
+        os.environ['TEMP'] = temp_dir
+        os.environ['TMP'] = temp_dir
+        tempfile.tempdir = temp_dir
+        print(f"✓ Set temp directory to: {temp_dir}")
+    
+    return temp_dir, upload_dir
+
+def create_app():
+    # Set up temp directories BEFORE creating Flask app
+    temp_dir, upload_dir = setup_temp_directories()
+    
+    app = Flask(__name__)
+    
+    # Setup custom logging configuration
+    try:
+        from logging_config import setup_logging, DEFAULT_LOG_LEVEL
+        setup_logging(os.environ.get('LOG_LEVEL', DEFAULT_LOG_LEVEL))
+    except ImportError:
+        pass  # Fallback to default logging if logging_config is not available
+    
+    # Configuration
+    app.config['SECRET_KEY'] = os.environ.get('SECRET_KEY', 'dev-secret-key')
+    app.config['JWT_SECRET_KEY'] = os.environ.get('JWT_SECRET_KEY', 'jwt-secret-key')
+    
+    # Fix strict slashes - this prevents 308 redirects for trailing slashes
+    app.url_map.strict_slashes = False
+    app.config['JWT_ACCESS_TOKEN_EXPIRES'] = 86400  # 24 hours
+    
+    # Set longer timeouts for AI operations
+    app.config['TIMEOUT'] = 300  # 5 minutes
+    app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024  # 16 MB max upload
+    
+    # Configure Flask/Werkzeug file upload settings
+    app.config['UPLOAD_FOLDER'] = upload_dir
+    app.config['UPLOAD_EXTENSIONS'] = ['.jpg', '.jpeg', '.png']
+    
+    # Configure temp directory for Flask/Werkzeug
+    if temp_dir and os.path.isdir(temp_dir):
+        app.config['TEMP_FOLDER'] = temp_dir
+        print(f"✓ Flask configured with temp directory: {temp_dir}")
+    
+    # Additional Werkzeug configuration for multipart form handling
+    app.config['MAX_CONTENT_PATH'] = None  # Don't limit content path
+    
+    # Configure Werkzeug to handle uploads without temp files for small files
+    app.config['MAX_FORM_MEMORY_SIZE'] = 16 * 1024 * 1024  # Keep small uploads in memory
+    
+    # Initialize extensions
+    CORS(app, resources={r"/api/*": {"origins": "*"}})
+    jwt = JWTManager(app)
+    
+    # Register blueprints
+    from app.routes.auth import auth_bp
+    from app.routes.personas import personas_bp
+    from app.routes.focus_groups import focus_groups_bp
+    from app.routes.ai_personas import ai_personas_bp
+    from app.routes.focus_group_ai import focus_group_ai_bp
+    
+    app.register_blueprint(auth_bp, url_prefix='/api/auth')
+    app.register_blueprint(personas_bp, url_prefix='/api/personas')
+    app.register_blueprint(focus_groups_bp, url_prefix='/api/focus-groups')
+    app.register_blueprint(ai_personas_bp, url_prefix='/api/ai-personas')
+    app.register_blueprint(focus_group_ai_bp, url_prefix='/api/focus-group-ai')
+    
+    # Health check endpoint
+    @app.route('/api/health', methods=['GET'])
+    def health_check():
+        return {'status': 'ok', 'message': 'Backend is running'}, 200
+    
+    return app

backend/app/db.py

@@ -0,0 +1,67 @@
+from pymongo import MongoClient
+import os
+import logging
+
+# MongoDB connection
+def get_db():
+    # Try to read environment variables for MongoDB credentials
+    mongo_user = os.environ.get('MONGO_USER')
+    mongo_pass = os.environ.get('MONGO_PASS')
+    mongo_host = os.environ.get('MONGO_HOST', 'localhost')
+    mongo_port = os.environ.get('MONGO_PORT', '27017')
+    
+    # Try with standard credentials first
+    standard_credentials = [
+        {"user": "admin", "pass": "admin", "db": "admin"},
+        {"user": "mongodb", "pass": "mongodb", "db": "admin"},
+        {"user": "root", "pass": "root", "db": "admin"},
+        {"user": "user", "pass": "pass", "db": "admin"}
+    ]
+    
+    # Try each set of standard credentials
+    for creds in standard_credentials:
+        try:
+            uri = f"mongodb://{creds['user']}:{creds['pass']}@{mongo_host}:{mongo_port}/semblance_db?authSource={creds['db']}"
+            client = MongoClient(uri, serverSelectionTimeoutMS=2000)
+            db = client.semblance_db
+            # Test the connection with a simple command
+            db.command('ping')
+            logging.debug(f"Successfully connected to MongoDB with standard credentials ({creds['user']})")
+            return db
+        except Exception as e:
+            # Continue trying other credentials
+            pass
+    
+    # Try to connect without authentication if standard credentials don't work
+    try:
+        client = MongoClient(f'mongodb://{mongo_host}:{mongo_port}', serverSelectionTimeoutMS=5000)
+        # Simply use the database as is - MongoDB will allow this if auth is not required
+        db = client.semblance_db
+        # Test the connection with a simple command
+        db.command('ping')
+        # Try a write operation to verify we have proper access
+        test_result = db.test_collection.insert_one({"test": "auth_test"})
+        db.test_collection.delete_one({"_id": test_result.inserted_id})
+        logging.debug("Successfully connected to MongoDB without authentication")
+        return db
+    except Exception as e:
+        logging.debug(f"Could not connect without auth: {e}")
+    
+    # If we get here, we need authentication - try with environment vars if provided
+    if mongo_user and mongo_pass:
+        try:
+            uri = f"mongodb://{mongo_user}:{mongo_pass}@{mongo_host}:{mongo_port}/semblance_db?authSource=admin"
+            client = MongoClient(uri, serverSelectionTimeoutMS=5000)
+            db = client.semblance_db
+            db.command('ping')  # Test the connection
+            logging.debug(f"Successfully connected to MongoDB with credentials for user: {mongo_user}")
+            return db
+        except Exception as e:
+            logging.warning(f"Failed to connect with environment credentials: {e}")
+    
+    # Last resort - log warning and return client that will fail later if DB actually needs auth
+    logging.warning("Could not authenticate with MongoDB. If authentication is required, operations will fail.")
+    logging.warning("To fix this: Set MONGO_USER and MONGO_PASS environment variables.")
+    # Return a client that will likely fail when operations are performed, but the app will start
+    client = MongoClient(f'mongodb://{mongo_host}:{mongo_port}', serverSelectionTimeoutMS=5000)
+    return client.semblance_db

backend/app/models/focus_group.py

@@ -0,0 +1,865 @@
+from bson import ObjectId
+from app.db import get_db
+from datetime import datetime
+import traceback
+import uuid
+import os
+
+class FocusGroup:
+    @staticmethod
+    def create(focus_group_data, user_id):
+        db = get_db()
+        
+        # Add metadata
+        focus_group_data["created_at"] = datetime.utcnow()
+        focus_group_data["created_by"] = user_id
+        
+        # Only set default status if not provided
+        if "status" not in focus_group_data:
+            focus_group_data["status"] = "new"
+        
+        result = db.focus_groups.insert_one(focus_group_data)
+        return str(result.inserted_id)
+    
+    @staticmethod
+    def find_by_id(focus_group_id):
+        db = get_db()
+        try:
+            focus_group = db.focus_groups.find_one({"_id": ObjectId(focus_group_id)})
+            if focus_group:
+                focus_group["_id"] = str(focus_group["_id"])
+            return focus_group
+        except:
+            return None
+    
+    @staticmethod
+    def find_by_user(user_id, limit=50):
+        db = get_db()
+        focus_groups = db.focus_groups.find({"created_by": user_id}).sort("created_at", -1).limit(limit)
+        result = []
+        
+        for group in focus_groups:
+            group["_id"] = str(group["_id"])
+            result.append(group)
+            
+        return result
+    
+    @staticmethod
+    def get_all(limit=50):
+        import logging
+        logger = logging.getLogger('app.focus_group_model')
+        
+        try:
+            logger.debug(f"=== FocusGroup.get_all() called with limit={limit} ===")
+            db = get_db()
+            logger.debug(f"Database connection obtained: {db}")
+            
+            # Check if collection exists and has data
+            collection = db.focus_groups
+            total_count = collection.count_documents({})
+            logger.debug(f"Total focus groups in database: {total_count}")
+            
+            focus_groups = list(db.focus_groups.find().sort("created_at", -1).limit(limit))
+            logger.debug(f"Query returned {len(focus_groups)} focus groups")
+            
+            result = []
+            for group in focus_groups:
+                group["_id"] = str(group["_id"])
+                result.append(group)
+                logger.debug(f"Processed group: {group.get('name', 'Unknown')} (ID: {group['_id']})")
+                
+            logger.debug(f"Returning {len(result)} processed focus groups")
+            return result
+        except Exception as e:
+            logger.error(f"Error in FocusGroup.get_all: {e}")
+            logger.exception("Full exception traceback:")
+            print(f"Error in FocusGroup.get_all: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def update(focus_group_id, data):
+        db = get_db()
+        
+        # Create a copy of the data to avoid modifying the original
+        filtered_data = data.copy()
+        
+        # Remove fields that shouldn't be updated
+        if '_id' in filtered_data:
+            del filtered_data['_id']
+        if 'id' in filtered_data:
+            del filtered_data['id']
+        if 'created_at' in filtered_data:
+            del filtered_data['created_at']
+        if 'created_by' in filtered_data:
+            del filtered_data['created_by']
+            
+        # Set the updated timestamp
+        filtered_data["updated_at"] = datetime.utcnow()
+        
+        result = db.focus_groups.update_one(
+            {"_id": ObjectId(focus_group_id)},
+            {"$set": filtered_data}
+        )
+        
+        return result.modified_count > 0
+    
+    @staticmethod
+    def _cleanup_focus_group_assets(focus_group_id, uploaded_assets):
+        """Clean up all creative asset files for a focus group."""
+        cleaned_files = []
+        failed_files = []
+        
+        if not uploaded_assets:
+            return cleaned_files, failed_files
+            
+        # Get upload folder paths (reuse existing logic from routes)
+        base_dir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))  # Go up to backend/
+        upload_dir = os.path.join(base_dir, 'uploads', f'focus-group-{focus_group_id}')
+        main_upload_dir = os.path.join(base_dir, 'uploads')
+        
+        for asset in uploaded_assets:
+            filename = asset.get('filename')
+            if not filename:
+                continue
+                
+            file_deleted = False
+            
+            try:
+                # Try subdirectory location first
+                subdirectory_path = os.path.join(upload_dir, filename)
+                if os.path.exists(subdirectory_path):
+                    os.remove(subdirectory_path)
+                    file_deleted = True
+                    cleaned_files.append(filename)
+                    print(f"Deleted asset file: {subdirectory_path}")
+                
+                # Try flat storage location if not found in subdirectory
+                if not file_deleted:
+                    flat_path = os.path.join(main_upload_dir, filename)
+                    if os.path.exists(flat_path):
+                        os.remove(flat_path)
+                        file_deleted = True
+                        cleaned_files.append(filename)
+                        print(f"Deleted asset file: {flat_path}")
+                
+                if not file_deleted:
+                    print(f"Warning: Asset file not found for deletion: {filename}")
+                    failed_files.append(filename)
+                        
+            except Exception as e:
+                print(f"Error deleting asset file {filename}: {e}")
+                failed_files.append(filename)
+        
+        # Try to remove empty subdirectory
+        try:
+            if os.path.exists(upload_dir) and not os.listdir(upload_dir):
+                os.rmdir(upload_dir)
+                print(f"Removed empty upload directory: {upload_dir}")
+        except Exception as e:
+            print(f"Warning: Could not remove upload directory {upload_dir}: {e}")
+        
+        return cleaned_files, failed_files
+    
+    @staticmethod
+    def _cleanup_focus_group_collections(focus_group_id):
+        """Clean up all related collection documents for a focus group."""
+        db = get_db()
+        cleaned_collections = []
+        failed_collections = []
+        
+        # Collections to clean up
+        collections_to_clean = [
+            ('focus_group_messages', 'focus_group_id'),
+            ('focus_group_themes', 'focus_group_id'), 
+            ('focus_group_notes', 'focus_group_id'),
+            ('focus_group_reasoning', 'focus_group_id'),
+            ('focus_group_mode_events', 'focus_group_id')
+        ]
+        
+        for collection_name, field_name in collections_to_clean:
+            try:
+                collection = getattr(db, collection_name)
+                result = collection.delete_many({field_name: focus_group_id})
+                if result.deleted_count > 0:
+                    cleaned_collections.append(f"{collection_name}: {result.deleted_count} documents")
+                    print(f"Cleaned up {result.deleted_count} documents from {collection_name}")
+                else:
+                    print(f"No documents found in {collection_name} for focus group {focus_group_id}")
+            except Exception as e:
+                print(f"Error cleaning up {collection_name}: {e}")
+                failed_collections.append(collection_name)
+        
+        return cleaned_collections, failed_collections
+
+    @staticmethod
+    def delete(focus_group_id):
+        """Delete a focus group and all its associated data including creative assets."""
+        db = get_db()
+        
+        try:
+            # First, get the focus group data to access uploaded assets
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                print(f"Focus group {focus_group_id} not found")
+                return False
+            
+            uploaded_assets = focus_group.get('uploaded_assets', [])
+            
+            # Clean up creative asset files
+            cleaned_files, failed_files = FocusGroup._cleanup_focus_group_assets(focus_group_id, uploaded_assets)
+            
+            # Clean up related collections
+            cleaned_collections, failed_collections = FocusGroup._cleanup_focus_group_collections(focus_group_id)
+            
+            # Finally, delete the main focus group document
+            result = db.focus_groups.delete_one({"_id": ObjectId(focus_group_id)})
+            
+            if result.deleted_count > 0:
+                print(f"Successfully deleted focus group {focus_group_id}")
+                print(f"Cleaned up {len(cleaned_files)} asset files: {cleaned_files}")
+                print(f"Cleaned up collections: {cleaned_collections}")
+                
+                if failed_files:
+                    print(f"Warning: Failed to delete some asset files: {failed_files}")
+                if failed_collections:
+                    print(f"Warning: Failed to clean some collections: {failed_collections}")
+                
+                return True
+            else:
+                print(f"Failed to delete focus group {focus_group_id} from database")
+                return False
+                
+        except Exception as e:
+            print(f"Error during focus group deletion: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def add_participant(focus_group_id, persona_id):
+        db = get_db()
+        result = db.focus_groups.update_one(
+            {"_id": ObjectId(focus_group_id)},
+            {"$addToSet": {"participants": persona_id}}
+        )
+        return result.modified_count > 0
+    
+    @staticmethod
+    def remove_participant(focus_group_id, persona_id):
+        db = get_db()
+        result = db.focus_groups.update_one(
+            {"_id": ObjectId(focus_group_id)},
+            {"$pull": {"participants": persona_id}}
+        )
+        return result.modified_count > 0
+        
+    @staticmethod
+    def get_messages(focus_group_id, limit=100):
+        """Get all messages for a focus group."""
+        db = get_db()
+        try:
+            # Get all messages and sort chronologically
+            messages = list(db.focus_group_messages.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("created_at", 1))
+            
+            # Convert ObjectId to strings
+            for message in messages:
+                if "_id" in message:
+                    message["_id"] = str(message["_id"])
+            
+            return messages
+        except Exception as e:
+            print(f"Error getting messages for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def add_message(focus_group_id, message_data):
+        """Add a new message to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+                
+            # Prepare the message
+            message = {
+                "focus_group_id": focus_group_id,
+                "text": message_data.get("text", ""),
+                "type": message_data.get("type", "response"),
+                "senderId": message_data.get("senderId", ""),
+                "created_at": datetime.utcnow(),
+                "highlighted": message_data.get("highlighted", False),
+                "attached_assets": message_data.get("attached_assets", []),  # List of asset filenames
+                "activates_visual_context": message_data.get("activates_visual_context", False)  # Visual context activation flag
+            }
+            
+            # Insert the message
+            result = db.focus_group_messages.insert_one(message)
+            
+            if result.inserted_id:
+                message_id = str(result.inserted_id)
+                
+                # If this message activates visual context, update the focus group's active visual context
+                if message.get("activates_visual_context") and message.get("attached_assets"):
+                    FocusGroup._activate_visual_assets(focus_group_id, message.get("attached_assets"), message_id)
+                
+                return message_id
+            else:
+                return None
+                
+        except Exception as e:
+            print(f"Error adding message to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return None
+            
+    @staticmethod
+    def update_message_highlight(focus_group_id, message_id, highlighted):
+        """Update the highlighted status of a message."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return False
+                
+            # Update the message
+            result = db.focus_group_messages.update_one(
+                {"_id": ObjectId(message_id), "focus_group_id": focus_group_id},
+                {"$set": {"highlighted": highlighted, "updated_at": datetime.utcnow()}}
+            )
+            
+            return result.modified_count > 0
+        except Exception as e:
+            print(f"Error updating message highlight in focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+
+    @staticmethod
+    def get_generated_themes(focus_group_id):
+        """Get all generated themes for a focus group."""
+        db = get_db()
+        try:
+            # Get themes associated with this focus group
+            themes = list(db.focus_group_themes.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("created_at", -1))
+            
+            # Convert ObjectId to strings
+            for theme in themes:
+                if "_id" in theme:
+                    theme["_id"] = str(theme["_id"])
+            
+            return themes
+        except Exception as e:
+            print(f"Error getting themes for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def add_generated_theme(focus_group_id, theme_data):
+        """Add a new generated theme to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+                
+            # Prepare the theme
+            theme = {
+                "focus_group_id": focus_group_id,
+                "id": theme_data.get("id", f"theme-{str(uuid.uuid4())}"),
+                "title": theme_data.get("title", ""),
+                "description": theme_data.get("description", ""),
+                "quotes": theme_data.get("quotes", []),
+                "created_at": datetime.utcnow(),
+                "source": "generated"
+            }
+            
+            # Insert the theme
+            result = db.focus_group_themes.insert_one(theme)
+            
+            # Return the id of the new theme
+            return str(result.inserted_id)
+        except Exception as e:
+            print(f"Error adding theme to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return None
+    
+    @staticmethod
+    def add_generated_themes(focus_group_id, themes_data):
+        """Add multiple generated themes to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+            
+            # Prepare the themes
+            themes = []
+            theme_ids = []
+            
+            for theme_data in themes_data:
+                theme_id = f"theme-{str(uuid.uuid4())}"
+                theme = {
+                    "focus_group_id": focus_group_id,
+                    "id": theme_id,
+                    "title": theme_data.get("title", ""),
+                    "description": theme_data.get("description", ""),
+                    "quotes": theme_data.get("quotes", []),
+                    "created_at": datetime.utcnow(),
+                    "source": "generated"
+                }
+                themes.append(theme)
+                theme_ids.append(theme_id)
+            
+            # Insert the themes
+            if themes:
+                result = db.focus_group_themes.insert_many(themes)
+                
+                # Return the ids of the new themes
+                return theme_ids
+            
+            return []
+        except Exception as e:
+            print(f"Error adding themes to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def delete_generated_theme(focus_group_id, theme_id):
+        """Delete a generated theme from a focus group."""
+        db = get_db()
+        try:
+            # Delete the theme
+            result = db.focus_group_themes.delete_one(
+                {"focus_group_id": focus_group_id, "id": theme_id}
+            )
+            
+            return result.deleted_count > 0
+        except Exception as e:
+            print(f"Error deleting theme {theme_id} from focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def get_reasoning_history(focus_group_id, limit=50):
+        """Get reasoning history for a focus group."""
+        db = get_db()
+        try:
+            # Get reasoning entries associated with this focus group
+            reasoning_entries = list(db.focus_group_reasoning.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("timestamp", -1).limit(limit))
+            
+            # Convert ObjectId to strings and format timestamps
+            for entry in reasoning_entries:
+                if "_id" in entry:
+                    entry["_id"] = str(entry["_id"])
+                # Ensure timestamp is in the expected format
+                if "timestamp" in entry and isinstance(entry["timestamp"], datetime):
+                    entry["timestamp"] = entry["timestamp"].isoformat()
+            
+            return reasoning_entries
+        except Exception as e:
+            print(f"Error getting reasoning history for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def add_reasoning_entry(focus_group_id, reasoning_data):
+        """Add a reasoning entry to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+                
+            # Prepare the reasoning entry
+            reasoning_entry = {
+                "focus_group_id": focus_group_id,
+                "timestamp": reasoning_data.get("timestamp", datetime.utcnow()),
+                "action": reasoning_data.get("action", "unknown"),
+                "reasoning": reasoning_data.get("reasoning", ""),
+                "details": reasoning_data.get("details", {}),
+                "execution_status": reasoning_data.get("execution_status", "pending"),
+                "execution_result": reasoning_data.get("execution_result", None),
+                "created_at": datetime.utcnow()
+            }
+            
+            # Convert timestamp string to datetime if needed
+            if isinstance(reasoning_entry["timestamp"], str):
+                try:
+                    reasoning_entry["timestamp"] = datetime.fromisoformat(reasoning_entry["timestamp"].replace('Z', '+00:00'))
+                except:
+                    reasoning_entry["timestamp"] = datetime.utcnow()
+            
+            # Insert the reasoning entry
+            result = db.focus_group_reasoning.insert_one(reasoning_entry)
+            
+            # Return the id of the new entry
+            return str(result.inserted_id)
+        except Exception as e:
+            print(f"Error adding reasoning entry to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return None
+    
+    @staticmethod
+    def update_reasoning_execution(focus_group_id, reasoning_id, execution_result):
+        """Update the execution result of a reasoning entry."""
+        db = get_db()
+        try:
+            # Update the reasoning entry
+            result = db.focus_group_reasoning.update_one(
+                {"_id": ObjectId(reasoning_id), "focus_group_id": focus_group_id},
+                {"$set": {
+                    "execution_status": "success" if not execution_result.get("error") else "error",
+                    "execution_result": execution_result,
+                    "updated_at": datetime.utcnow()
+                }}
+            )
+            
+            return result.modified_count > 0
+        except Exception as e:
+            print(f"Error updating reasoning execution in focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def get_notes(focus_group_id, limit=100):
+        """Get all notes for a focus group."""
+        db = get_db()
+        try:
+            # Look for a notes collection associated with this focus group
+            notes = list(db.focus_group_notes.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("created_at", -1).limit(limit))
+            
+            # Convert ObjectId to strings
+            for note in notes:
+                if "_id" in note:
+                    note["_id"] = str(note["_id"])
+                    # Set id field to match frontend expectations
+                    note["id"] = note["_id"]
+            
+            return notes
+        except Exception as e:
+            print(f"Error getting notes for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def add_note(focus_group_id, note_data):
+        """Add a new note to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+                
+            # Prepare the note
+            note = {
+                "focus_group_id": focus_group_id,
+                "content": note_data.get("content", ""),
+                "associatedMessageId": note_data.get("associatedMessageId"),
+                "sectionInfo": note_data.get("sectionInfo", {}),
+                "elapsedTime": note_data.get("elapsedTime", 0),
+                "timestamp": note_data.get("timestamp", datetime.utcnow().isoformat()),
+                "created_at": datetime.utcnow(),
+                "createdAt": datetime.utcnow()
+            }
+            
+            # Convert timestamp string to datetime if needed
+            if isinstance(note["timestamp"], str):
+                try:
+                    note["timestamp"] = datetime.fromisoformat(note["timestamp"].replace('Z', '+00:00'))
+                except:
+                    note["timestamp"] = datetime.utcnow()
+            
+            # Insert the note
+            result = db.focus_group_notes.insert_one(note)
+            
+            # Return the id of the new note
+            return str(result.inserted_id)
+        except Exception as e:
+            print(f"Error adding note to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return None
+    
+    @staticmethod
+    def delete_note(focus_group_id, note_id):
+        """Delete a note from a focus group."""
+        db = get_db()
+        try:
+            # Delete the note
+            result = db.focus_group_notes.delete_one(
+                {"_id": ObjectId(note_id), "focus_group_id": focus_group_id}
+            )
+            
+            return result.deleted_count > 0
+        except Exception as e:
+            print(f"Error deleting note {note_id} from focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def add_mode_event(focus_group_id, event_type, user_id=None):
+        """Add a mode switch event to a focus group."""
+        db = get_db()
+        try:
+            # Ensure the focus group exists
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return None
+                
+            # Prepare the mode event
+            mode_event = {
+                "focus_group_id": focus_group_id,
+                "event_type": event_type,  # 'ai_mode_started' or 'manual_mode_started'
+                "timestamp": datetime.utcnow(),
+                "user_id": user_id,  # None for system-initiated changes
+                "created_at": datetime.utcnow()
+            }
+            
+            # Insert the mode event
+            result = db.focus_group_mode_events.insert_one(mode_event)
+            
+            # Return the id of the new mode event
+            return str(result.inserted_id)
+        except Exception as e:
+            print(f"Error adding mode event to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return None
+    
+    @staticmethod
+    def get_mode_events(focus_group_id, limit=100):
+        """Get all mode events for a focus group."""
+        db = get_db()
+        try:
+            # Look for mode events associated with this focus group
+            mode_events = list(db.focus_group_mode_events.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("timestamp", 1).limit(limit))
+            
+            # Convert ObjectId to strings
+            for event in mode_events:
+                if "_id" in event:
+                    event["_id"] = str(event["_id"])
+                    # Set id field to match frontend expectations
+                    event["id"] = event["_id"]
+            
+            return mode_events
+        except Exception as e:
+            print(f"Error getting mode events for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def add_uploaded_assets(focus_group_id, assets_metadata):
+        """Add uploaded asset metadata to a focus group."""
+        db = get_db()
+        try:
+            # Clean the metadata to remove file_path before storing in DB
+            cleaned_assets = []
+            for asset in assets_metadata:
+                cleaned_asset = {
+                    "filename": asset["filename"],
+                    "original_name": asset["original_name"],
+                    "size": asset["size"],
+                    "mime_type": asset["mime_type"],
+                    "upload_date": asset["upload_date"]
+                }
+                cleaned_assets.append(cleaned_asset)
+            
+            # Add assets to the focus group
+            result = db.focus_groups.update_one(
+                {"_id": ObjectId(focus_group_id)},
+                {
+                    "$push": {"uploaded_assets": {"$each": cleaned_assets}},
+                    "$set": {"updated_at": datetime.utcnow()}
+                }
+            )
+            
+            return result.modified_count > 0
+        except Exception as e:
+            print(f"Error adding uploaded assets to focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def remove_uploaded_asset(focus_group_id, filename):
+        """Remove an uploaded asset metadata from a focus group."""
+        db = get_db()
+        try:
+            # Remove asset from the focus group
+            result = db.focus_groups.update_one(
+                {"_id": ObjectId(focus_group_id)},
+                {
+                    "$pull": {"uploaded_assets": {"filename": filename}},
+                    "$set": {"updated_at": datetime.utcnow()}
+                }
+            )
+            
+            return result.modified_count > 0
+        except Exception as e:
+            print(f"Error removing uploaded asset from focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def get_uploaded_assets(focus_group_id):
+        """Get uploaded assets for a focus group."""
+        db = get_db()
+        try:
+            focus_group = db.focus_groups.find_one({"_id": ObjectId(focus_group_id)})
+            if focus_group:
+                return focus_group.get('uploaded_assets', [])
+            return []
+        except Exception as e:
+            print(f"Error getting uploaded assets for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def _activate_visual_assets(focus_group_id, asset_filenames, message_id):
+        """Internal method to activate visual assets in conversation context."""
+        db = get_db()
+        try:
+            # Get current message count to determine sequence number
+            message_count = db.focus_group_messages.count_documents({"focus_group_id": focus_group_id})
+            
+            # Get existing visual context to check for duplicate assets
+            focus_group = db.focus_groups.find_one({"_id": ObjectId(focus_group_id)})
+            existing_context = focus_group.get("active_visual_context", []) if focus_group else []
+            
+            # Track which assets are new vs existing
+            new_records = []
+            updated_filenames = []
+            
+            for filename in asset_filenames:
+                # Check if this asset is already in the active context
+                existing_asset = next((asset for asset in existing_context if asset["filename"] == filename), None)
+                
+                if existing_asset:
+                    # Asset already exists - we'll update its sequence to current position
+                    updated_filenames.append(filename)
+                    print(f"🔄 Re-activating existing visual asset: {filename} (moving to sequence {message_count})")
+                else:
+                    # New asset - add to records
+                    new_records.append({
+                        "filename": filename,
+                        "activated_at_message_id": message_id,
+                        "activated_at_sequence": message_count,
+                        "activation_timestamp": datetime.utcnow()
+                    })
+                    print(f"🆕 Activating new visual asset: {filename} at sequence {message_count}")
+            
+            # First, update existing assets to current sequence
+            for filename in updated_filenames:
+                db.focus_groups.update_one(
+                    {"_id": ObjectId(focus_group_id), "active_visual_context.filename": filename},
+                    {
+                        "$set": {
+                            "active_visual_context.$.activated_at_message_id": message_id,
+                            "active_visual_context.$.activated_at_sequence": message_count,
+                            "active_visual_context.$.activation_timestamp": datetime.utcnow(),
+                            "updated_at": datetime.utcnow()
+                        }
+                    }
+                )
+            
+            # Then, add any new assets
+            result = None
+            if new_records:
+                result = db.focus_groups.update_one(
+                    {"_id": ObjectId(focus_group_id)},
+                    {
+                        "$push": {"active_visual_context": {"$each": new_records}},
+                        "$set": {"updated_at": datetime.utcnow()}
+                    },
+                    upsert=True
+                )
+            else:
+                # If we only updated existing assets, just set the updated_at timestamp
+                result = db.focus_groups.update_one(
+                    {"_id": ObjectId(focus_group_id)},
+                    {"$set": {"updated_at": datetime.utcnow()}}
+                )
+            
+            print(f"🎨 Activated visual assets for focus group {focus_group_id}: {asset_filenames}")
+            return True
+            
+        except Exception as e:
+            print(f"Error activating visual assets for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False
+    
+    @staticmethod
+    def get_active_visual_context(focus_group_id):
+        """Get all images that are active in conversation context for this focus group."""
+        db = get_db()
+        try:
+            focus_group = db.focus_groups.find_one({"_id": ObjectId(focus_group_id)})
+            if focus_group:
+                return focus_group.get('active_visual_context', [])
+            return []
+        except Exception as e:
+            print(f"Error getting active visual context for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def get_messages_with_visual_context(focus_group_id, limit=100):
+        """Get messages with enhanced visual context information."""
+        db = get_db()
+        try:
+            # Get all messages
+            messages = list(db.focus_group_messages.find(
+                {"focus_group_id": focus_group_id}
+            ).sort("created_at", 1))
+            
+            # Convert ObjectId to strings and add sequence numbers
+            for i, message in enumerate(messages):
+                if "_id" in message:
+                    message["_id"] = str(message["_id"])
+                message["sequence"] = i + 1
+                
+                # Add flag indicating if this message has visual context available
+                active_context = FocusGroup.get_active_visual_context(focus_group_id)
+                message["has_visual_context"] = any(
+                    asset["activated_at_sequence"] <= message["sequence"] 
+                    for asset in active_context
+                )
+            
+            return messages
+        except Exception as e:
+            print(f"Error getting messages with visual context for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return []
+    
+    @staticmethod
+    def clear_visual_context(focus_group_id):
+        """Clear all active visual context for a focus group (useful for testing)."""
+        db = get_db()
+        try:
+            result = db.focus_groups.update_one(
+                {"_id": ObjectId(focus_group_id)},
+                {
+                    "$unset": {"active_visual_context": ""},
+                    "$set": {"updated_at": datetime.utcnow()}
+                }
+            )
+            
+            print(f"🧹 Cleared visual context for focus group {focus_group_id}")
+            return result.modified_count > 0
+            
+        except Exception as e:
+            print(f"Error clearing visual context for focus group {focus_group_id}: {e}")
+            print(traceback.format_exc())
+            return False

backend/app/models/persona.py

@@ -0,0 +1,123 @@
+from bson import ObjectId
+from app.db import get_db
+from datetime import datetime
+
+class Persona:
+    @staticmethod
+    def create(persona_data, user_id=None):
+        db = get_db()
+        
+        # Add metadata
+        persona_data["created_at"] = datetime.utcnow()
+        persona_data["created_by"] = user_id
+        
+        result = db.personas.insert_one(persona_data)
+        return str(result.inserted_id)
+    
+    @staticmethod
+    def find_by_id(persona_id):
+        db = get_db()
+        try:
+            # If persona_id is already an ObjectId, use it directly
+            if isinstance(persona_id, ObjectId):
+                object_id = persona_id
+            else:
+                try:
+                    # Try to convert to ObjectId
+                    object_id = ObjectId(persona_id)
+                except Exception as e:
+                    print(f"Invalid ObjectId format: {persona_id}, error: {e}")
+                    # Try lookup by string ID as fallback
+                    persona = db.personas.find_one({"id": persona_id})
+                    if persona:
+                        persona["_id"] = str(persona["_id"])
+                        return persona
+                    return None
+            
+            # Lookup by ObjectId
+            persona = db.personas.find_one({"_id": object_id})
+            if persona:
+                persona["_id"] = str(persona["_id"])
+            return persona
+        except Exception as e:
+            print(f"Error in find_by_id: {e}, persona_id: {persona_id}")
+            return None
+    
+    @staticmethod
+    def find_by_user(user_id, limit=100):
+        db = get_db()
+        personas = db.personas.find({"created_by": user_id}).sort("created_at", -1).limit(limit)
+        result = []
+        
+        for persona in personas:
+            persona["_id"] = str(persona["_id"])
+            result.append(persona)
+            
+        return result
+    
+    @staticmethod
+    def get_all(limit=100):
+        try:
+            db = get_db()
+            personas = list(db.personas.find().sort("created_at", -1).limit(limit))
+            result = []
+            
+            for persona in personas:
+                persona["_id"] = str(persona["_id"])
+                result.append(persona)
+                
+            return result
+        except Exception as e:
+            print(f"Error in Persona.get_all: {e}")
+            return []
+    
+    @staticmethod
+    def update(persona_id, data):
+        db = get_db()
+        
+        # Create a copy of the data to avoid modifying the original
+        filtered_data = data.copy()
+        
+        # Remove fields that shouldn't be updated
+        if '_id' in filtered_data:
+            del filtered_data['_id']
+        if 'id' in filtered_data:
+            del filtered_data['id']
+        if 'created_at' in filtered_data:
+            del filtered_data['created_at']
+        if 'created_by' in filtered_data:
+            del filtered_data['created_by']
+            
+        # Set the updated timestamp
+        filtered_data["updated_at"] = datetime.utcnow()
+        
+        result = db.personas.update_one(
+            {"_id": ObjectId(persona_id)},
+            {"$set": filtered_data}
+        )
+        
+        return result.modified_count > 0
+    
+    @staticmethod
+    def delete(persona_id):
+        db = get_db()
+        try:
+            # If persona_id is already an ObjectId, use it directly
+            if isinstance(persona_id, ObjectId):
+                object_id = persona_id
+            else:
+                try:
+                    # Try to convert to ObjectId
+                    object_id = ObjectId(persona_id)
+                except Exception as e:
+                    print(f"Invalid ObjectId format for delete: {persona_id}, error: {e}")
+                    # Try delete by string ID as fallback
+                    result = db.personas.delete_one({"id": persona_id})
+                    return result.deleted_count > 0
+            
+            # Delete by ObjectId
+            result = db.personas.delete_one({"_id": object_id})
+            return result.deleted_count > 0
+        except Exception as e:
+            print(f"Error in delete: {e}, persona_id: {persona_id}")
+            return False

backend/app/models/user.py

@@ -0,0 +1,90 @@
+import bcrypt
+from bson import ObjectId
+from app.db import get_db
+
+class User:
+    def __init__(self, username, email, password_hash, role="user"):
+        self.username = username
+        self.email = email
+        self.password_hash = password_hash
+        self.role = role
+    
+    @staticmethod
+    def hash_password(password):
+        salt = bcrypt.gensalt()
+        hashed = bcrypt.hashpw(password.encode('utf-8'), salt)
+        return hashed.decode('utf-8')
+    
+    @staticmethod
+    def check_password(password_hash, password):
+        return bcrypt.checkpw(password.encode('utf-8'), password_hash.encode('utf-8'))
+    
+    @staticmethod
+    def find_by_username(username):
+        db = get_db()
+        user_data = db.users.find_one({"username": username})
+        return user_data
+    
+    @staticmethod
+    def find_by_email(email):
+        db = get_db()
+        user_data = db.users.find_one({"email": email})
+        return user_data
+    
+    @staticmethod
+    def find_by_id(user_id):
+        db = get_db()
+        user_data = db.users.find_one({"_id": ObjectId(user_id)})
+        return user_data
+    
+    def to_dict(self):
+        return {
+            "username": self.username,
+            "email": self.email,
+            "role": self.role
+        }
+    
+    def save(self):
+        db = get_db()
+        user_data = {
+            "username": self.username,
+            "email": self.email,
+            "password_hash": self.password_hash,
+            "role": self.role
+        }
+        result = db.users.insert_one(user_data)
+        return result.inserted_id
+    
+    @staticmethod
+    def create_default_user():
+        try:
+            db = get_db()
+            
+            # First check if users collection exists
+            collections = db.list_collection_names()
+            if "users" not in collections:
+                print("Creating users collection")
+                db.create_collection("users")
+            
+            # Safely check if user exists, handling potential auth errors
+            try:
+                user_exists = db.users.count_documents({"username": "user"}) > 0
+            except Exception as e:
+                print(f"Error checking for default user: {e}")
+                # If we can't query, assume we need to create the user
+                user_exists = False
+            
+            if not user_exists:
+                default_user = User(
+                    username="user",
+                    email="user@example.com",
+                    password_hash=User.hash_password("pass"),
+                    role="admin"
+                )
+                default_user.save()
+                print("Default user created successfully")
+            else:
+                print("Default user already exists")
+        except Exception as e:
+            print(f"Error creating default user: {e}")
+            # Don't raise the exception - allow the app to continue even if we can't create the user

backend/app/routes/ai_personas.py

@@ -0,0 +1,947 @@
+"""
+AI Persona Generation Routes.
+These endpoints handle the generation of synthetic personas using AI models.
+"""
+
+from flask import Blueprint, request, jsonify, current_app
+from flask_jwt_extended import jwt_required, get_jwt_identity
+import time
+import concurrent.futures
+from werkzeug.serving import is_running_from_reloader
+
+from app.services.ai_persona_service import (
+    generate_persona, 
+    customize_persona_prompt,
+    generate_basic_personas,
+    generate_persona_summary,
+    generate_persona_download_summary,
+    enhance_audience_brief,
+    PersonaGenerationError
+)
+from app.services.customer_data_service import customer_data_service, CustomerDataServiceError
+from app.models.persona import Persona
+
+# Get timeout for AI requests
+AI_REQUEST_TIMEOUT = 300  # 5 minutes in seconds
+
+ai_personas_bp = Blueprint('ai_personas', __name__)
+
+
+@ai_personas_bp.route('/generate-basic-profiles', methods=['POST'])
+@jwt_required()
+def generate_basic_profiles():
+    """
+    First stage of the two-stage persona generation process.
+    
+    This endpoint generates basic demographic and psychographic profiles 
+    based on a research brief. These profiles can then be used as input
+    for generating complete personas in the second stage.
+    
+    Request body:
+    {
+        "audience_brief": "A detailed description of the audience context...",
+        "count": 5,  # Number of profiles to generate (default 5)
+        "temperature": 0.8  # Optional, controls randomness in generation
+    }
+    
+    Returns:
+        A JSON object containing an array of basic persona profiles
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    audience_brief = data.get('audience_brief')
+    if not audience_brief:
+        return jsonify({"error": "Missing audience brief", "message": "Audience brief is required"}), 400
+    
+    research_objective = data.get('research_objective')  # Optional parameter
+        
+    count = data.get('count', 5)
+    if count < 1 or count > 10:  # Limit the number for performance reasons
+        return jsonify({"error": "Invalid count", "message": "Count must be between 1 and 10"}), 400
+    
+    temperature = data.get('temperature', 0.8)
+    if not (0 <= temperature <= 1):
+        temperature = 0.8
+    
+    customer_data_session_id = data.get('customer_data_session_id')  # Optional parameter
+    
+    try:
+        # Generate basic profiles
+        basic_profiles = generate_basic_personas(
+            audience_brief=audience_brief,
+            research_objective=research_objective,
+            count=count,
+            temperature=temperature,
+            customer_data_session_id=customer_data_session_id
+        )
+        
+        return jsonify({
+            "message": f"Successfully generated {len(basic_profiles)} basic persona profiles",
+            "profiles": basic_profiles
+        }), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Basic profiles generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate basic profiles", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in basic profiles generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/complete-persona', methods=['POST'])
+@jwt_required()
+def complete_persona():
+    """
+    Second stage of the two-stage persona generation process.
+    
+    This endpoint takes a basic persona profile and expands it into a complete persona
+    with goals, frustrations, motivations, and more detailed characteristics.
+    
+    Request body:
+    {
+        "basic_profile": {
+            "name": "John Smith",
+            "age": "30-35",
+            ...
+        },
+        "temperature": 0.7  # Optional, controls randomness in generation
+    }
+    
+    Returns:
+        A JSON object containing the complete persona
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    basic_profile = data.get('basic_profile')
+    if not basic_profile:
+        return jsonify({"error": "Missing basic profile", "message": "Basic profile is required"}), 400
+    
+    temperature = data.get('temperature', 0.7)
+    if not (0 <= temperature <= 1):
+        temperature = 0.7
+    
+    try:
+        # Complete the persona
+        complete_persona_data = generate_persona(
+            basic_persona=basic_profile,
+            temperature=temperature
+        )
+        
+        return jsonify({
+            "message": "Successfully completed persona generation",
+            "persona": complete_persona_data
+        }), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Complete persona generation error: {str(e)}")
+        return jsonify({"error": "Failed to complete persona", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in complete persona generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/complete-and-save-persona', methods=['POST'])
+@jwt_required()
+def complete_and_save_persona():
+    """
+    Second stage of the two-stage persona generation process that also saves the
+    persona to the database.
+    
+    This endpoint takes a basic persona profile, expands it into a complete persona,
+    and saves it to the database.
+    
+    Request body:
+    {
+        "basic_profile": {
+            "name": "John Smith",
+            "age": "30-35",
+            ...
+        },
+        "temperature": 0.7,  # Optional, controls randomness in generation
+        "customer_data_session_id": "uuid"  # Optional, session ID for customer data
+    }
+    
+    Returns:
+        A JSON object containing the complete persona with its database ID
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    basic_profile = data.get('basic_profile')
+    if not basic_profile:
+        return jsonify({"error": "Missing basic profile", "message": "Basic profile is required"}), 400
+    
+    temperature = data.get('temperature', 0.7)
+    if not (0 <= temperature <= 1):
+        temperature = 0.7
+    
+    customer_data_session_id = data.get('customer_data_session_id')  # Optional parameter
+    
+    try:
+        # Complete the persona
+        complete_persona_data = generate_persona(
+            basic_persona=basic_profile,
+            temperature=temperature,
+            customer_data_session_id=customer_data_session_id
+        )
+        
+        # Generate AI summary for the persona
+        try:
+            summary_data = generate_persona_summary(
+                persona_data=complete_persona_data,
+                temperature=temperature
+            )
+            
+            # Add summary fields to the persona data
+            complete_persona_data['aiSynthesizedBio'] = summary_data['aiSynthesizedBio']
+            complete_persona_data['qualitativeAttributes'] = summary_data['qualitativeAttributes']
+            complete_persona_data['topPersonalityTraits'] = summary_data['topPersonalityTraits']
+            
+            print(f"Generated summary for persona {complete_persona_data.get('name', 'Unknown')}")
+            
+        except Exception as summary_error:
+            # Log the error but don't fail the entire persona creation
+            current_app.logger.warning(f"Failed to generate summary for persona: {str(summary_error)}")
+            print(f"Warning: Could not generate summary for persona: {str(summary_error)}")
+        
+        # Remove the generated ID as the database will assign one
+        if 'id' in complete_persona_data:
+            del complete_persona_data['id']
+        
+        # Save to database
+        persona_id = Persona.create(complete_persona_data, user_id)
+        
+        # Add the database ID to the response
+        complete_persona_data['_id'] = str(persona_id)
+        
+        return jsonify({
+            "message": "Successfully completed and saved persona",
+            "persona": complete_persona_data,
+            "persona_id": str(persona_id)
+        }), 201
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Complete and save persona error: {str(e)}")
+        return jsonify({"error": "Failed to complete and save persona", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in complete and save persona: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/generate', methods=['POST'])
+@jwt_required()
+def generate_ai_persona():
+    """
+    Generate a synthetic persona using AI and return it without saving.
+    
+    Request body can include customization parameters:
+    {
+        "age_range": "25-34",
+        "gender": "Female",
+        "occupation_type": "Technology",
+        "education_level": "Bachelor's Degree",
+        "location_type": "Urban US",
+        "personality_traits": "introverted, analytical",
+        "interests": "technology, reading",
+        "temperature": 0.7
+    }
+    
+    Returns:
+        A JSON object containing the generated persona
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    try:
+        # Extract customization parameters
+        customization = customize_persona_prompt(
+            age_range=data.get('age_range'),
+            gender=data.get('gender'),
+            occupation_type=data.get('occupation_type'),
+            education_level=data.get('education_level'),
+            location_type=data.get('location_type'),
+            personality_traits=data.get('personality_traits'),
+            interests=data.get('interests'),
+            audience_brief=data.get('audience_brief')
+        )
+        
+        # Set temperature if provided
+        temperature = data.get('temperature', 0.7)
+        if not (0 <= temperature <= 1):
+            temperature = 0.7
+        
+        # Generate the persona
+        persona_data = generate_persona(
+            prompt_customization=customization,
+            temperature=temperature
+        )
+        
+        return jsonify(persona_data), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"AI Persona generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate persona", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in persona generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/generate-and-save', methods=['POST'])
+@jwt_required()
+def generate_and_save_persona():
+    """
+    Generate a synthetic persona using AI and save it to the database.
+    
+    Request body can include customization parameters as in the generate endpoint.
+    
+    Returns:
+        A JSON object containing the generated and saved persona, including its database ID
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    try:
+        # Extract customization parameters
+        customization = customize_persona_prompt(
+            age_range=data.get('age_range'),
+            gender=data.get('gender'),
+            occupation_type=data.get('occupation_type'),
+            education_level=data.get('education_level'),
+            location_type=data.get('location_type'),
+            personality_traits=data.get('personality_traits'),
+            interests=data.get('interests'),
+            audience_brief=data.get('audience_brief')
+        )
+        
+        # Set temperature if provided
+        temperature = data.get('temperature', 0.7)
+        if not (0 <= temperature <= 1):
+            temperature = 0.7
+        
+        # Generate the persona
+        persona_data = generate_persona(
+            prompt_customization=customization,
+            temperature=temperature
+        )
+        
+        # Generate AI summary for the persona
+        try:
+            summary_data = generate_persona_summary(
+                persona_data=persona_data,
+                temperature=temperature
+            )
+            
+            # Add summary fields to the persona data
+            persona_data['aiSynthesizedBio'] = summary_data['aiSynthesizedBio']
+            persona_data['qualitativeAttributes'] = summary_data['qualitativeAttributes']
+            persona_data['topPersonalityTraits'] = summary_data['topPersonalityTraits']
+            
+            print(f"Generated summary for persona {persona_data.get('name', 'Unknown')}")
+            
+        except Exception as summary_error:
+            # Log the error but don't fail the entire persona creation
+            current_app.logger.warning(f"Failed to generate summary for persona: {str(summary_error)}")
+            print(f"Warning: Could not generate summary for persona: {str(summary_error)}")
+        
+        # Remove the generated ID as the database will assign one
+        if 'id' in persona_data:
+            del persona_data['id']
+        
+        # Save to database
+        persona_id = Persona.create(persona_data, user_id)
+        
+        # Add the database ID to the response
+        persona_data['_id'] = str(persona_id)
+        
+        return jsonify({
+            "message": "AI persona generated and saved successfully",
+            "persona": persona_data,
+            "persona_id": str(persona_id)
+        }), 201
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"AI Persona generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate persona", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in persona generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/batch-generate', methods=['POST'])
+@jwt_required()
+def batch_generate_personas():
+    """
+    Generate multiple synthetic personas using AI.
+    
+    Request body:
+    {
+        "count": 5,  # Number of personas to generate
+        "customizations": [
+            {
+                "age_range": "25-34",
+                "gender": "Female",
+                ...  
+            },
+            ...  # Optional list of customizations, one per persona
+        ],
+        "temperature": 0.7  # Optional, applies to all generations
+    }
+    
+    Returns:
+        A JSON object containing an array of generated personas
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    count = data.get('count', 1)
+    if count < 1 or count > 10:  # Limit the number for performance reasons
+        return jsonify({"error": "Invalid count", "message": "Count must be between 1 and 10"}), 400
+    
+    customizations = data.get('customizations', [])
+    temperature = data.get('temperature', 0.7)
+    
+    try:
+        # Prepare customization prompts for each persona
+        generation_tasks = []
+        for i in range(count):
+            # Use a customization if available for this index
+            custom_prompt = None
+            if i < len(customizations):
+                custom_data = customizations[i]
+                custom_prompt = customize_persona_prompt(
+                    age_range=custom_data.get('age_range'),
+                    gender=custom_data.get('gender'),
+                    occupation_type=custom_data.get('occupation_type'),
+                    education_level=custom_data.get('education_level'),
+                    location_type=custom_data.get('location_type'),
+                    personality_traits=custom_data.get('personality_traits'),
+                    interests=custom_data.get('interests'),
+                    audience_brief=custom_data.get('audience_brief')
+                )
+            
+            # Add to the list of tasks to be executed in parallel
+            generation_tasks.append({
+                'prompt_customization': custom_prompt,
+                'temperature': temperature
+            })
+            
+        # Generate personas in parallel
+        personas = []
+        with concurrent.futures.ThreadPoolExecutor(max_workers=min(count, 4)) as executor:
+            # Start the generation tasks
+            future_to_task = {
+                executor.submit(
+                    generate_persona, 
+                    task['prompt_customization'], 
+                    None,  # No basic_persona for this endpoint
+                    task['temperature']
+                ): i for i, task in enumerate(generation_tasks)
+            }
+            
+            # Process completed tasks as they finish
+            for future in concurrent.futures.as_completed(future_to_task):
+                try:
+                    persona_data = future.result()
+                    personas.append(persona_data)
+                except Exception as exc:
+                    current_app.logger.error(f"Persona generation task failed with error: {exc}")
+                    raise PersonaGenerationError(f"Failed to generate one of the personas: {str(exc)}")
+        
+        return jsonify({
+            "message": f"Successfully generated {len(personas)} personas in parallel",
+            "personas": personas
+        }), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"AI Persona batch generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate personas", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in batch persona generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/batch-generate-and-save', methods=['POST'])
+@jwt_required()
+def batch_generate_and_save_personas():
+    """
+    Generate multiple synthetic personas using AI and save them to the database.
+    
+    Request body format is the same as batch-generate endpoint.
+    
+    Returns:
+        A JSON object containing the array of generated and saved personas with their IDs
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    count = data.get('count', 1)
+    if count < 1 or count > 10:  # Limit for performance
+        return jsonify({"error": "Invalid count", "message": "Count must be between 1 and 10"}), 400
+    
+    customizations = data.get('customizations', [])
+    temperature = data.get('temperature', 0.7)
+    
+    try:
+        # Prepare customization prompts for each persona
+        generation_tasks = []
+        for i in range(count):
+            # Use a customization if available for this index
+            custom_prompt = None
+            if i < len(customizations):
+                custom_data = customizations[i]
+                custom_prompt = customize_persona_prompt(
+                    age_range=custom_data.get('age_range'),
+                    gender=custom_data.get('gender'),
+                    occupation_type=custom_data.get('occupation_type'),
+                    education_level=custom_data.get('education_level'),
+                    location_type=custom_data.get('location_type'),
+                    personality_traits=custom_data.get('personality_traits'),
+                    interests=custom_data.get('interests'),
+                    audience_brief=custom_data.get('audience_brief')
+                )
+            
+            # Add to the list of tasks to be executed in parallel
+            generation_tasks.append({
+                'prompt_customization': custom_prompt,
+                'temperature': temperature
+            })
+        
+        # Generate personas in parallel
+        generated_personas = []
+        with concurrent.futures.ThreadPoolExecutor(max_workers=min(count, 4)) as executor:
+            # Start the generation tasks
+            future_to_task = {
+                executor.submit(
+                    generate_persona, 
+                    task['prompt_customization'], 
+                    None,  # No basic_persona for this endpoint
+                    task['temperature']
+                ): i for i, task in enumerate(generation_tasks)
+            }
+            
+            # Process completed tasks as they finish
+            for future in concurrent.futures.as_completed(future_to_task):
+                try:
+                    persona_data = future.result()
+                    generated_personas.append(persona_data)
+                except Exception as exc:
+                    current_app.logger.error(f"Persona generation task failed with error: {exc}")
+                    raise PersonaGenerationError(f"Failed to generate one of the personas: {str(exc)}")
+        
+        # Save all generated personas to the database
+        personas = []
+        persona_ids = []
+        for persona_data in generated_personas:
+            # Generate AI summary for each persona  
+            try:
+                summary_data = generate_persona_summary(
+                    persona_data=persona_data,
+                    temperature=temperature
+                )
+                
+                # Add summary fields to the persona data
+                persona_data['aiSynthesizedBio'] = summary_data['aiSynthesizedBio']
+                persona_data['qualitativeAttributes'] = summary_data['qualitativeAttributes']
+                persona_data['topPersonalityTraits'] = summary_data['topPersonalityTraits']
+                
+                print(f"Generated summary for persona {persona_data.get('name', 'Unknown')}")
+                
+            except Exception as summary_error:
+                # Log the error but don't fail the entire persona creation
+                current_app.logger.warning(f"Failed to generate summary for persona: {str(summary_error)}")
+                print(f"Warning: Could not generate summary for persona: {str(summary_error)}")
+            
+            # Remove generated ID before saving
+            if 'id' in persona_data:
+                del persona_data['id']
+                
+            # Save to database
+            persona_id = Persona.create(persona_data, user_id)
+            
+            # Add database ID to the response
+            persona_data['_id'] = str(persona_id)
+            personas.append(persona_data)
+            persona_ids.append(str(persona_id))
+        
+        return jsonify({
+            "message": f"Successfully generated and saved {len(personas)} personas in parallel",
+            "personas": personas,
+            "persona_ids": persona_ids
+        }), 201
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"AI Persona batch generation and save error: {str(e)}")
+        return jsonify({"error": "Failed to generate and save personas", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in batch persona generation and save: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/generate-persona-summary', methods=['POST'])
+@jwt_required()
+def generate_summary_for_persona():
+    """
+    Generate an AI-synthesized summary for an existing persona.
+    
+    This endpoint takes a complete persona and generates a concise summary
+    containing an AI-synthesized bio, qualitative attributes, and top personality traits
+    for display on persona cards.
+    
+    Request body:
+    {
+        "persona_data": {
+            "name": "John Smith",
+            "age": "30-35",
+            "occupation": "Software Engineer",
+            ...  // Complete persona object
+        },
+        "temperature": 0.7  // Optional, controls randomness in generation
+    }
+    
+    Returns:
+        A JSON object containing the generated summary data
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    persona_data = data.get('persona_data')
+    if not persona_data:
+        return jsonify({"error": "Missing persona data", "message": "Persona data is required"}), 400
+    
+    # Validate that persona_data has required fields
+    required_fields = ["name", "age", "gender", "occupation", "personality"]
+    missing_fields = [field for field in required_fields if field not in persona_data]
+    if missing_fields:
+        return jsonify({
+            "error": "Invalid persona data", 
+            "message": f"Persona data is missing required fields: {', '.join(missing_fields)}"
+        }), 400
+    
+    temperature = data.get('temperature', 0.7)
+    if not (0 <= temperature <= 1):
+        temperature = 0.7
+    
+    try:
+        # Generate the summary
+        summary_data = generate_persona_summary(
+            persona_data=persona_data,
+            temperature=temperature
+        )
+        
+        return jsonify({
+            "message": "Successfully generated persona summary",
+            "summary": summary_data
+        }), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Persona summary generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate persona summary", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in persona summary generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/enhance-audience-brief', methods=['POST'])
+@jwt_required()
+def enhance_audience_brief_endpoint():
+    """
+    Generate suggestions to improve an audience brief for better persona generation.
+    
+    This endpoint analyzes an audience brief from a behavioral science perspective
+    and provides actionable suggestions to make it more comprehensive and useful
+    for creating detailed, realistic personas.
+    
+    Request body:
+    {
+        "audience_brief": "A detailed description of the audience context...",
+        "research_objective": "The specific research goals and questions...",
+        "temperature": 0.7  # Optional, controls randomness in generation
+    }
+    
+    Returns:
+        A JSON object containing separate suggestion arrays for audience_brief and research_objective
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    audience_brief = data.get('audience_brief')
+    research_objective = data.get('research_objective')
+    
+    if not audience_brief:
+        return jsonify({"error": "Missing audience brief", "message": "Audience brief is required"}), 400
+    if not research_objective:
+        return jsonify({"error": "Missing research objective", "message": "Research objective is required"}), 400
+        
+    # Validate both fields have minimum length (10 characters each as per requirements)
+    if len(audience_brief.strip()) < 10:
+        return jsonify({"error": "Audience brief too short", "message": "Audience brief must be at least 10 characters long"}), 400
+    if len(research_objective.strip()) < 10:
+        return jsonify({"error": "Research objective too short", "message": "Research objective must be at least 10 characters long"}), 400
+    
+    temperature = data.get('temperature', 0.7)
+    if not (0 <= temperature <= 1):
+        temperature = 0.7
+    
+    try:
+        # Generate enhancement suggestions
+        suggestions = enhance_audience_brief(
+            audience_brief=audience_brief.strip(),
+            research_objective=research_objective.strip(),
+            temperature=temperature
+        )
+        
+        total_count = len(suggestions['audience_brief']) + len(suggestions['research_objective'])
+        return jsonify({
+            "message": f"Successfully generated {total_count} enhancement suggestions",
+            "suggestions": suggestions
+        }), 200
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Audience brief enhancement error: {str(e)}")
+        return jsonify({"error": "Failed to enhance audience brief", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in audience brief enhancement: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/batch-generate-summaries', methods=['POST'])
+@jwt_required()
+def batch_generate_summaries():
+    """
+    Generate comprehensive markdown summaries for multiple personas for download/client review.
+    
+    This endpoint takes a list of persona IDs, fetches their complete data, and generates
+    detailed summaries using LLM processing. Personas are processed in parallel batches of 10
+    to optimize performance while staying within API limits.
+    
+    Request body:
+    {
+        "persona_ids": ["id1", "id2", "id3", ...],  # Array of persona IDs to summarize
+        "temperature": 0.7  # Optional, controls randomness in generation
+    }
+    
+    Returns:
+        A JSON object containing the generated summaries and any errors encountered
+    """
+    user_id = get_jwt_identity()
+    data = request.get_json() or {}
+    
+    # Extract parameters
+    persona_ids = data.get('persona_ids', [])
+    if not persona_ids:
+        return jsonify({"error": "Missing persona IDs", "message": "At least one persona ID is required"}), 400
+    
+    if not isinstance(persona_ids, list):
+        return jsonify({"error": "Invalid persona IDs", "message": "persona_ids must be an array"}), 400
+    
+    if len(persona_ids) > 50:  # Reasonable limit for batch processing
+        return jsonify({"error": "Too many personas", "message": "Maximum 50 personas can be processed at once"}), 400
+    
+    temperature = data.get('temperature', 0.7)
+    if not (0 <= temperature <= 1):
+        temperature = 0.7
+    
+    try:
+        # Fetch all persona data first
+        personas_data = []
+        missing_personas = []
+        
+        for persona_id in persona_ids:
+            try:
+                persona = Persona.find_by_id(persona_id)
+                if persona:
+                    personas_data.append(persona)
+                else:
+                    missing_personas.append(persona_id)
+            except Exception as e:
+                current_app.logger.warning(f"Failed to fetch persona {persona_id}: {str(e)}")
+                missing_personas.append(persona_id)
+        
+        if not personas_data:
+            return jsonify({
+                "error": "No valid personas found", 
+                "message": "None of the provided persona IDs could be found"
+            }), 404
+        
+        # Process personas in batches of 10
+        batch_size = 10
+        successful_summaries = []
+        failed_summaries = []
+        
+        def process_persona_summary(persona_data):
+            """Helper function to process a single persona summary"""
+            try:
+                summary = generate_persona_download_summary(
+                    persona_data=persona_data,
+                    temperature=temperature
+                )
+                return {
+                    'success': True,
+                    'persona_id': persona_data.get('_id', persona_data.get('id')),
+                    'persona_name': persona_data.get('name', 'Unknown'),
+                    'summary': summary
+                }
+            except Exception as e:
+                return {
+                    'success': False,
+                    'persona_id': persona_data.get('_id', persona_data.get('id')),
+                    'persona_name': persona_data.get('name', 'Unknown'),
+                    'error': str(e)
+                }
+        
+        # Process in batches
+        for i in range(0, len(personas_data), batch_size):
+            batch = personas_data[i:i + batch_size]
+            current_app.logger.info(f"Processing batch {i//batch_size + 1}: {len(batch)} personas")
+            
+            # Process this batch in parallel
+            with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as executor:
+                # Submit all tasks for this batch
+                future_to_persona = {
+                    executor.submit(process_persona_summary, persona): persona 
+                    for persona in batch
+                }
+                
+                # Collect results as they complete
+                for future in concurrent.futures.as_completed(future_to_persona):
+                    result = future.result()
+                    if result['success']:
+                        successful_summaries.append(result)
+                    else:
+                        failed_summaries.append(result)
+                        current_app.logger.error(f"Failed to generate summary for persona {result['persona_name']}: {result['error']}")
+        
+        # Prepare response
+        total_requested = len(persona_ids)
+        total_found = len(personas_data)
+        total_successful = len(successful_summaries)
+        total_failed = len(failed_summaries) + len(missing_personas)
+        
+        response_data = {
+            "message": f"Processed {total_successful} of {total_requested} personas successfully",
+            "summary_stats": {
+                "total_requested": total_requested,
+                "total_found": total_found,
+                "total_successful": total_successful,
+                "total_failed": total_failed,
+                "missing_personas": len(missing_personas)
+            },
+            "summaries": successful_summaries
+        }
+        
+        # Add error details if there were failures
+        if failed_summaries or missing_personas:
+            response_data["errors"] = {
+                "failed_summaries": failed_summaries,
+                "missing_personas": missing_personas
+            }
+        
+        # Determine appropriate status code
+        if total_successful == 0:
+            return jsonify(response_data), 500  # Complete failure
+        elif total_successful < total_requested:
+            return jsonify(response_data), 206  # Partial success
+        else:
+            return jsonify(response_data), 200  # Complete success
+        
+    except PersonaGenerationError as e:
+        current_app.logger.error(f"Batch summary generation error: {str(e)}")
+        return jsonify({"error": "Failed to generate summaries", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in batch summary generation: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/upload-customer-data', methods=['POST'])
+@jwt_required()
+def upload_customer_data():
+    """
+    Upload customer data files and parse them using LlamaParse.
+    
+    Request: multipart/form-data with 'files' containing the customer data files
+    
+    Returns:
+        JSON object with session_id for the uploaded and parsed files
+    """
+    user_id = get_jwt_identity()
+    
+    try:
+        current_app.logger.debug(f"=== UPLOAD CUSTOMER DATA API called for user {user_id} ===")
+        
+        # Check if files were provided
+        if 'files' not in request.files:
+            current_app.logger.warning("No 'files' key in request.files")
+            return jsonify({"error": "No files provided"}), 400
+        
+        files = request.files.getlist('files')
+        if not files or all(f.filename == '' for f in files):
+            current_app.logger.warning("No files selected")
+            return jsonify({"error": "No files selected"}), 400
+        
+        current_app.logger.info(f"Processing {len(files)} customer data files")
+        
+        # Upload and parse files using customer data service
+        session_id = customer_data_service.upload_and_parse_files(files)
+        
+        current_app.logger.info(f"Successfully processed customer data files with session_id: {session_id}")
+        
+        return jsonify({
+            "success": True,
+            "session_id": session_id,
+            "message": f"Successfully processed {len(files)} files"
+        }), 200
+        
+    except CustomerDataServiceError as e:
+        current_app.logger.error(f"Customer data service error: {str(e)}")
+        return jsonify({"error": "Failed to process customer data", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in upload customer data: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500
+
+
+@ai_personas_bp.route('/cleanup-customer-data/<session_id>', methods=['DELETE'])
+@jwt_required()
+def cleanup_customer_data(session_id):
+    """
+    Clean up customer data files for a specific session.
+    
+    Args:
+        session_id: The session ID for the customer data to clean up
+    
+    Returns:
+        JSON object confirming successful cleanup
+    """
+    user_id = get_jwt_identity()
+    
+    try:
+        current_app.logger.debug(f"=== CLEANUP CUSTOMER DATA API called for session {session_id} by user {user_id} ===")
+        
+        # Clean up files using customer data service
+        success = customer_data_service.cleanup_session(session_id)
+        
+        if success:
+            current_app.logger.info(f"Successfully cleaned up customer data for session: {session_id}")
+            return jsonify({
+                "success": True,
+                "message": f"Successfully cleaned up customer data for session {session_id}"
+            }), 200
+        else:
+            current_app.logger.warning(f"No customer data found to clean up for session: {session_id}")
+            return jsonify({
+                "success": True,
+                "message": f"No customer data found for session {session_id} (may have been already cleaned up)"
+            }), 200
+        
+    except CustomerDataServiceError as e:
+        current_app.logger.error(f"Customer data service error during cleanup: {str(e)}")
+        return jsonify({"error": "Failed to cleanup customer data", "message": str(e)}), 500
+    except Exception as e:
+        current_app.logger.error(f"Unexpected error in cleanup customer data: {str(e)}")
+        return jsonify({"error": "Internal server error", "message": "An unexpected error occurred"}), 500

backend/app/routes/auth.py

@@ -0,0 +1,143 @@
+from flask import Blueprint, request, jsonify
+from flask_jwt_extended import create_access_token, jwt_required, get_jwt_identity
+from app.models.user import User
+
+auth_bp = Blueprint('auth', __name__)
+
+@auth_bp.route('/register', methods=['POST'])
+def register():
+    data = request.get_json()
+    
+    if not data or not data.get('username') or not data.get('email') or not data.get('password'):
+        return jsonify({"message": "Missing required fields"}), 400
+    
+    username = data.get('username')
+    email = data.get('email')
+    password = data.get('password')
+    
+    # Check if user already exists
+    if User.find_by_username(username):
+        return jsonify({"message": "Username already taken"}), 409
+    if User.find_by_email(email):
+        return jsonify({"message": "Email already registered"}), 409
+    
+    # Create new user
+    hashed_password = User.hash_password(password)
+    new_user = User(username=username, email=email, password_hash=hashed_password)
+    user_id = new_user.save()
+    
+    # Generate access token
+    access_token = create_access_token(identity=str(user_id))
+    
+    return jsonify({
+        "message": "User registered successfully",
+        "access_token": access_token,
+        "user": new_user.to_dict()
+    }), 201
+
+@auth_bp.route('/login', methods=['POST'])
+def login():
+    try:
+        data = request.get_json()
+        
+        if not data or not data.get('username') or not data.get('password'):
+            return jsonify({"message": "Missing username or password"}), 400
+        
+        username = data.get('username')
+        password = data.get('password')
+        
+        # Default credentials for development/testing
+        if username == "user" and password == "pass":
+            # Create a mock user with a valid ObjectId
+            from bson import ObjectId
+            default_id = str(ObjectId())
+            
+            user_mock = {
+                "_id": default_id,
+                "username": "user",
+                "email": "user@example.com",
+                "role": "admin"
+            }
+            
+            # Generate access token
+            access_token = create_access_token(identity=default_id)
+            
+            return jsonify({
+                "message": "Login successful (default user)",
+                "access_token": access_token,
+                "user": {
+                    "username": user_mock['username'],
+                    "email": user_mock['email'],
+                    "role": user_mock['role']
+                }
+            }), 200
+        
+        # Try to find user in database
+        try:
+            # Find user by username
+            user_data = User.find_by_username(username)
+            if not user_data:
+                return jsonify({"message": "Invalid username or password"}), 401
+            
+            # Check password
+            if not User.check_password(user_data['password_hash'], password):
+                return jsonify({"message": "Invalid username or password"}), 401
+            
+            # Generate access token
+            access_token = create_access_token(identity=str(user_data['_id']))
+            
+            return jsonify({
+                "message": "Login successful",
+                "access_token": access_token,
+                "user": {
+                    "username": user_data['username'],
+                    "email": user_data['email'],
+                    "role": user_data.get('role', 'user')
+                }
+            }), 200
+        except Exception as e:
+            print(f"Database error during login: {e}")
+            # If we can't access the database but it's the default user, still allow login
+            if username == "user" and password == "pass":
+                # This was handled above
+                pass
+            else:
+                return jsonify({"message": "Database error, please try again later"}), 500
+            
+    except Exception as e:
+        print(f"Unexpected error in login route: {e}")
+        return jsonify({"message": "Internal server error"}), 500
+
+@auth_bp.route('/me', methods=['GET'])
+@jwt_required()
+def get_profile():
+    user_id = get_jwt_identity()
+    
+    # Handle the default_id case specially
+    if user_id == "default_id":
+        # Return mock user data for default_id
+        return jsonify({
+            "username": "user",
+            "email": "user@example.com",
+            "role": "admin"
+        }), 200
+    
+    try:
+        user_data = User.find_by_id(user_id)
+        
+        if not user_data:
+            return jsonify({"message": "User not found"}), 404
+        
+        return jsonify({
+            "username": user_data['username'],
+            "email": user_data['email'],
+            "role": user_data.get('role', 'user')
+        }), 200
+    except Exception as e:
+        print(f"Error in get_profile: {e}")
+        # If there's an error, still return default user data
+        return jsonify({
+            "username": "user",
+            "email": "user@example.com",
+            "role": "user"
+        }), 200

backend/app/routes/personas.py

@@ -0,0 +1,153 @@
+from flask import Blueprint, request, jsonify
+from flask_jwt_extended import jwt_required, get_jwt_identity
+from app.models.persona import Persona
+from bson import ObjectId
+import datetime
+
+# Helper function to make MongoDB documents JSON serializable
+def make_serializable(obj):
+    if isinstance(obj, dict):
+        return {k: make_serializable(v) for k, v in obj.items()}
+    elif isinstance(obj, list):
+        return [make_serializable(item) for item in obj]
+    elif isinstance(obj, ObjectId):
+        return str(obj)
+    elif isinstance(obj, datetime.datetime):
+        return obj.isoformat()
+    else:
+        return obj
+
+personas_bp = Blueprint('personas', __name__)
+
+@personas_bp.route('', methods=['GET'])
+@personas_bp.route('/', methods=['GET'])
+@jwt_required(optional=True)  # Make JWT optional for development
+def get_personas():
+    try:
+        user_id = get_jwt_identity()
+        if user_id:
+            # If authenticated, get user's personas
+            personas = Persona.find_by_user(user_id)
+        else:
+            # For development, return all personas if not authenticated
+            personas = Persona.get_all()
+        
+        # Make personas serializable
+        serializable_personas = make_serializable(personas)
+        return jsonify(serializable_personas), 200
+    except Exception as e:
+        print(f"Error in get_personas: {e}")
+        return jsonify({"error": str(e)}), 500
+
+@personas_bp.route('/all', methods=['GET'])
+@jwt_required(optional=True)  # Make JWT optional for development
+def get_all_personas():
+    try:
+        personas = Persona.get_all()
+        # Make personas serializable
+        serializable_personas = make_serializable(personas)
+        return jsonify(serializable_personas), 200
+    except Exception as e:
+        print(f"Error in get_all_personas: {e}")
+        return jsonify({"error": str(e)}), 500
+
+@personas_bp.route('/<persona_id>', methods=['GET'])
+@jwt_required(optional=True)  # Make JWT optional for development
+def get_persona(persona_id):
+    try:
+        persona = Persona.find_by_id(persona_id)
+        if not persona:
+            return jsonify({"message": "Persona not found"}), 404
+            
+        # Make persona serializable
+        serializable_persona = make_serializable(persona)
+        return jsonify(serializable_persona), 200
+    except Exception as e:
+        print(f"Error in get_persona: {e}")
+        return jsonify({"error": str(e)}), 500
+
+@personas_bp.route('', methods=['POST'])
+@personas_bp.route('/', methods=['POST'])
+@jwt_required()
+def create_persona():
+    user_id = get_jwt_identity()
+    data = request.get_json()
+    
+    if not data:
+        return jsonify({"message": "No data provided"}), 400
+    
+    persona_id = Persona.create(data, user_id)
+    
+    return jsonify({
+        "message": "Persona created successfully",
+        "persona_id": persona_id
+    }), 201
+
+@personas_bp.route('/<persona_id>', methods=['PUT'])
+@jwt_required()
+def update_persona(persona_id):
+    try:
+        data = request.get_json()
+        
+        if not data:
+            return jsonify({"message": "No data provided"}), 400
+        
+        persona = Persona.find_by_id(persona_id)
+        if not persona:
+            return jsonify({"message": "Persona not found"}), 404
+        
+        # Ensure _id is not being modified
+        if '_id' in data:
+            del data['_id']
+        
+        # Ensure id is not being used for update
+        if 'id' in data:
+            del data['id']
+        
+        success = Persona.update(persona_id, data)
+        
+        if success:
+            # Get the updated persona and return it
+            updated_persona = Persona.find_by_id(persona_id)
+            return jsonify({
+                "message": "Persona updated successfully",
+                "persona": make_serializable(updated_persona)
+            }), 200
+        else:
+            return jsonify({"message": "No changes made to persona"}), 200
+    except Exception as e:
+        print(f"Error updating persona: {e}")
+        return jsonify({"message": f"Failed to update persona: {str(e)}"}), 500
+
+@personas_bp.route('/<persona_id>', methods=['DELETE'])
+@jwt_required()
+def delete_persona(persona_id):
+    persona = Persona.find_by_id(persona_id)
+    if not persona:
+        return jsonify({"message": "Persona not found"}), 404
+    
+    success = Persona.delete(persona_id)
+    
+    if success:
+        return jsonify({"message": "Persona deleted successfully"}), 200
+    else:
+        return jsonify({"message": "Failed to delete persona"}), 500
+
+@personas_bp.route('/batch', methods=['POST'])
+@jwt_required()
+def create_multiple_personas():
+    user_id = get_jwt_identity()
+    data = request.get_json()
+    
+    if not data or not isinstance(data, list):
+        return jsonify({"message": "Invalid data format. Expected list of personas"}), 400
+    
+    persona_ids = []
+    for persona_data in data:
+        persona_id = Persona.create(persona_data, user_id)
+        persona_ids.append(persona_id)
+    
+    return jsonify({
+        "message": f"Successfully created {len(persona_ids)} personas",
+        "persona_ids": persona_ids
+    }), 201

backend/app/services/ai_moderator_service.py

@@ -0,0 +1,770 @@
+"""
+AI Moderator Service
+This service handles AI-powered moderation of focus group discussions,
+including sequential navigation through structured discussion guides.
+"""
+
+from typing import Dict, List, Any, Optional, Tuple
+from flask import current_app
+from app.models.focus_group import FocusGroup
+from app.services.llm_service import LLMService, LLMServiceError
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+from app.utils.discussion_guide_schema import DiscussionGuideValidator, StructuredDiscussionGuide
+import json
+
+
+class AIModeratorService:
+    """Service for AI-powered focus group moderation."""
+    
+    @staticmethod
+    def _count_total_items(sections: List[Dict[str, Any]]) -> int:
+        """
+        Count the total number of questions and activities across all sections and subsections.
+        
+        Args:
+            sections: List of discussion guide sections
+            
+        Returns:
+            Total count of all questions and activities
+        """
+        total_count = 0
+        
+        for section in sections:
+            # Count activities in main section
+            if section.get('activities'):
+                total_count += len(section['activities'])
+            
+            # Count questions in main section
+            if section.get('questions'):
+                total_count += len(section['questions'])
+            
+            # Count items in subsections
+            if section.get('subsections'):
+                for subsection in section['subsections']:
+                    if subsection.get('activities'):
+                        total_count += len(subsection['activities'])
+                    if subsection.get('questions'):
+                        total_count += len(subsection['questions'])
+        
+        return total_count
+    
+    @staticmethod
+    def _count_completed_items(sections: List[Dict[str, Any]], moderator_position: Dict[str, Any]) -> int:
+        """
+        Count the number of completed questions and activities up to the current moderator position.
+        
+        Args:
+            sections: List of discussion guide sections
+            moderator_position: Current moderator position with section_index, item_index, item_type
+            
+        Returns:
+            Count of completed items
+        """
+        current_section_index = moderator_position.get('section_index', 0)
+        current_item_index = moderator_position.get('item_index', 0)
+        current_item_type = moderator_position.get('item_type', 'activity')
+        current_subsection_index = moderator_position.get('subsection_index', None)
+        
+        completed_count = 0
+        
+        # Special case: if we're past all sections, everything is completed
+        if current_section_index >= len(sections):
+            return AIModeratorService._count_total_items(sections)
+        
+        for section_idx, section in enumerate(sections):
+            if section_idx < current_section_index:
+                # All items in previous sections are completed
+                if section.get('activities'):
+                    completed_count += len(section['activities'])
+                if section.get('questions'):
+                    completed_count += len(section['questions'])
+                
+                # Count all items in subsections of previous sections
+                if section.get('subsections'):
+                    for subsection in section['subsections']:
+                        if subsection.get('activities'):
+                            completed_count += len(subsection['activities'])
+                        if subsection.get('questions'):
+                            completed_count += len(subsection['questions'])
+                            
+            elif section_idx == current_section_index:
+                # Current section - count items up to current position
+                if current_subsection_index is None:
+                    # Working at section level (not in a subsection)
+                    # Count all completed activities first, then questions up to current position
+                    activities = section.get('activities', [])
+                    questions = section.get('questions', [])
+                    
+                    if current_item_type == 'activity':
+                        # Currently on an activity - count completed activities up to current position
+                        # If item_index is past all activities, all activities are completed
+                        completed_count += min(current_item_index, len(activities))
+                        # If we're past all activities, also include all questions as completed
+                        if current_item_index >= len(activities):
+                            completed_count += len(questions)
+                    elif current_item_type == 'question':
+                        # Currently on a question - all activities are done, count questions up to current position
+                        completed_count += len(activities)
+                        # If item_index is past all questions, all questions are completed
+                        completed_count += min(current_item_index, len(questions))
+                else:
+                    # Working within a subsection
+                    # All section-level items are completed
+                    if section.get('activities'):
+                        completed_count += len(section['activities'])
+                    if section.get('questions'):
+                        completed_count += len(section['questions'])
+                    
+                    # Count completed subsection items
+                    subsections = section.get('subsections', [])
+                    for subsection_idx, subsection in enumerate(subsections):
+                        if subsection_idx < current_subsection_index:
+                            # All items in previous subsections are completed
+                            if subsection.get('activities'):
+                                completed_count += len(subsection['activities'])
+                            if subsection.get('questions'):
+                                completed_count += len(subsection['questions'])
+                        elif subsection_idx == current_subsection_index:
+                            # Current subsection - count items up to current position
+                            activities = subsection.get('activities', [])
+                            questions = subsection.get('questions', [])
+                            
+                            if current_item_type == 'activity':
+                                completed_count += min(current_item_index, len(activities))
+                                # If we're past all activities in subsection, also count all questions
+                                if current_item_index >= len(activities):
+                                    completed_count += len(questions)
+                            elif current_item_type == 'question':
+                                completed_count += len(activities)
+                                completed_count += min(current_item_index, len(questions))
+            
+            # Sections after current section are not yet completed, so we don't count them
+        
+        return completed_count
+    
+    @staticmethod
+    def get_moderator_status(focus_group_id: str) -> Dict[str, Any]:
+        """
+        Get the current moderator status for a focus group.
+        
+        Args:
+            focus_group_id: The focus group ID
+            
+        Returns:
+            Dictionary containing current moderator status
+        """
+        try:
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            # Get current moderator position
+            moderator_position = focus_group.get('moderator_position')
+            
+            # If no moderator position exists, initialize it
+            if not moderator_position:
+                moderator_position = {
+                    'section_index': 0,
+                    'item_index': 0,
+                    'item_type': 'activity'  # or 'question'
+                }
+                
+                # Save the initial position to the database
+                try:
+                    FocusGroup.update(focus_group_id, {
+                        'moderator_position': moderator_position
+                    })
+                    print(f"Initialized moderator position for focus group {focus_group_id}")
+                except Exception as e:
+                    print(f"Warning: Failed to initialize moderator position in database: {e}")
+            else:
+                # Reduce log verbosity - only log when position actually changes
+                if not hasattr(AIModeratorService, '_last_logged_positions'):
+                    AIModeratorService._last_logged_positions = {}
+                
+                last_position = AIModeratorService._last_logged_positions.get(focus_group_id)
+                if last_position != moderator_position:
+                    print(f"📍 Moderator position for focus group {focus_group_id}: {moderator_position}")
+                    AIModeratorService._last_logged_positions[focus_group_id] = moderator_position
+            
+            # Get discussion guide
+            discussion_guide = focus_group.get('discussionGuide', {})
+            
+            # If it's a string (old format), return basic status
+            if isinstance(discussion_guide, str):
+                return {
+                    "current_section": "Unknown",
+                    "current_item": "Discussion in progress",
+                    "progress": 0,
+                    "total_sections": 0,
+                    "legacy_format": True
+                }
+            
+            # Handle structured JSON format
+            if not discussion_guide or 'sections' not in discussion_guide:
+                return {"error": "No discussion guide found"}
+            
+            sections = discussion_guide['sections']
+            current_section_index = moderator_position.get('section_index', 0)
+            current_item_index = moderator_position.get('item_index', 0)
+            
+            # Validate indices
+            if current_section_index >= len(sections):
+                current_section_index = len(sections) - 1
+                moderator_position['section_index'] = current_section_index
+            
+            current_section = sections[current_section_index]
+            
+            # Get current item details
+            current_item = AIModeratorService._get_current_item(
+                current_section, current_item_index, moderator_position.get('item_type', 'activity')
+            )
+            
+            # Calculate granular progress based on individual questions/activities
+            total_items = AIModeratorService._count_total_items(sections)
+            completed_items = AIModeratorService._count_completed_items(sections, moderator_position)
+            
+            # Calculate progress percentage
+            progress = (completed_items / total_items * 100) if total_items > 0 else 0
+            
+            return {
+                "current_section": current_section.get('title', 'Unknown'),
+                "current_section_id": current_section.get('id', ''),
+                "current_item": current_item.get('content', 'No content') if current_item else 'End of section',
+                "current_item_id": current_item.get('id', '') if current_item else '',
+                "current_item_type": moderator_position.get('item_type', 'activity'),
+                "progress": progress,
+                "section_progress": current_item_index,
+                "total_sections": len(sections),
+                "moderator_position": moderator_position,
+                "section_type": current_section.get('type', 'unknown'),
+                "legacy_format": False
+            }
+            
+        except Exception as e:
+            return {"error": f"Error getting moderator status: {str(e)}"}
+    
+    @staticmethod
+    def advance_discussion(focus_group_id: str) -> Dict[str, Any]:
+        """
+        Advance the discussion to the next item in the guide and generate appropriate moderator response.
+        
+        Args:
+            focus_group_id: The focus group ID
+            
+        Returns:
+            Dictionary containing the moderator response and updated position
+        """
+        try:
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            # Get discussion guide
+            discussion_guide = focus_group.get('discussionGuide', {})
+            
+            # Handle legacy markdown format
+            if isinstance(discussion_guide, str):
+                return AIModeratorService._handle_legacy_advance(focus_group_id, discussion_guide)
+            
+            # Handle structured JSON format
+            if not discussion_guide or 'sections' not in discussion_guide:
+                return {"error": "No discussion guide found"}
+            
+            # Get current position
+            moderator_position = focus_group.get('moderator_position', {
+                'section_index': 0,
+                'item_index': 0,
+                'item_type': 'activity'
+            })
+            
+            # Advance to next item
+            new_position, next_item, section_info = AIModeratorService._advance_position(
+                discussion_guide, moderator_position
+            )
+            
+            if not next_item:
+                return {
+                    "message": "Discussion guide completed",
+                    "moderator_response": "Thank you everyone for your participation. We have covered all the topics in our discussion guide. This concludes our focus group session.",
+                    "position": new_position,
+                    "completed": True
+                }
+            
+            # Generate moderator response based on the next item
+            moderator_response = AIModeratorService._generate_moderator_response(
+                focus_group_id, next_item, section_info, new_position
+            )
+            
+            # Update focus group with new position
+            print(f"🎯 Advancing moderator position for focus group {focus_group_id}: {new_position}")
+            update_success = FocusGroup.update(focus_group_id, {
+                'moderator_position': new_position
+            })
+            
+            if update_success:
+                print(f"✅ Successfully updated moderator position in database")
+            else:
+                print(f"❌ Failed to update moderator position in database")
+            
+            return {
+                "message": "Discussion advanced successfully",
+                "moderator_response": moderator_response,
+                "position": new_position,
+                "current_item": next_item,
+                "section_info": section_info,
+                "completed": False
+            }
+            
+        except Exception as e:
+            return {"error": f"Error advancing discussion: {str(e)}"}
+    
+    @staticmethod
+    def set_moderator_position(focus_group_id: str, section_id: str, item_id: Optional[str] = None) -> Dict[str, Any]:
+        """
+        Set the moderator position to a specific section and item.
+        
+        Args:
+            focus_group_id: The focus group ID
+            section_id: The section ID to navigate to
+            item_id: The specific item ID (optional)
+            
+        Returns:
+            Dictionary containing the result and new position
+        """
+        try:
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            discussion_guide = focus_group.get('discussionGuide', {})
+            
+            if isinstance(discussion_guide, str):
+                return {"error": "Manual positioning not supported for legacy format"}
+            
+            if not discussion_guide or 'sections' not in discussion_guide:
+                return {"error": "No discussion guide found"}
+            
+            # Find the section
+            sections = discussion_guide['sections']
+            section_index = None
+            target_section = None
+            
+            for i, section in enumerate(sections):
+                if section.get('id') == section_id:
+                    section_index = i
+                    target_section = section
+                    break
+            
+            if section_index is None:
+                return {"error": f"Section '{section_id}' not found"}
+            
+            # Find the item if specified
+            item_index = 0
+            item_type = 'activity'
+            subsection_index = None
+            
+            if item_id:
+                # Look for the item in activities first, then questions
+                found = False
+                
+                # Check activities
+                if target_section.get('activities'):
+                    for i, activity in enumerate(target_section['activities']):
+                        if activity.get('id') == item_id:
+                            item_index = i
+                            item_type = 'activity'
+                            found = True
+                            break
+                
+                # Check questions if not found in activities
+                if not found and target_section.get('questions'):
+                    for i, question in enumerate(target_section['questions']):
+                        if question.get('id') == item_id:
+                            item_index = i
+                            item_type = 'question'
+                            found = True
+                            break
+                
+                # Check subsections if not found in main section
+                if not found and target_section.get('subsections'):
+                    for subsection_idx, subsection in enumerate(target_section['subsections']):
+                        # Check activities in current subsection
+                        if subsection.get('activities'):
+                            for i, activity in enumerate(subsection['activities']):
+                                if activity.get('id') == item_id:
+                                    subsection_index = subsection_idx
+                                    item_index = i
+                                    item_type = 'activity'
+                                    found = True
+                                    current_app.logger.info(f"📍 Found item '{item_id}' in subsection {subsection_idx} activity {i}, using subsection_index={subsection_index}, item_index={item_index}")
+                                    break
+                            if found:
+                                break
+                        
+                        # Check questions in subsection if not found in activities
+                        if not found and subsection.get('questions'):
+                            for i, question in enumerate(subsection['questions']):
+                                if question.get('id') == item_id:
+                                    subsection_index = subsection_idx
+                                    item_index = i
+                                    item_type = 'question'
+                                    found = True
+                                    current_app.logger.info(f"📍 Found item '{item_id}' in subsection {subsection_idx} question {i}, using subsection_index={subsection_index}, item_index={item_index}")
+                                    break
+                        
+                        if found:
+                            break
+                
+                if not found:
+                    return {"error": f"Item '{item_id}' not found in section '{section_id}'"}
+            
+            # Set new position
+            new_position = {
+                'section_index': section_index,
+                'item_index': item_index,
+                'item_type': item_type
+            }
+            
+            # Add subsection_index if item is found in a subsection
+            if subsection_index is not None:
+                new_position['subsection_index'] = subsection_index
+            
+            # Log detailed position information for debugging
+            if subsection_index is not None:
+                current_app.logger.info(f"🎯 Setting moderator position: section_index={section_index}, subsection_index={subsection_index}, item_index={item_index}, item_type={item_type}")
+            else:
+                current_app.logger.info(f"🎯 Setting moderator position: section_index={section_index}, item_index={item_index}, item_type={item_type}")
+            
+            # Update focus group
+            FocusGroup.update(focus_group_id, {
+                'moderator_position': new_position
+            })
+            
+            return {
+                "message": "Moderator position updated successfully",
+                "position": new_position,
+                "section_title": target_section.get('title', 'Unknown'),
+                "section_id": section_id
+            }
+            
+        except Exception as e:
+            return {"error": f"Error setting moderator position: {str(e)}"}
+    
+    @staticmethod
+    def _get_current_item(section: Dict[str, Any], item_index: int, item_type: str) -> Optional[Dict[str, Any]]:
+        """Get the current item from a section."""
+        if item_type == 'activity' and section.get('activities'):
+            activities = section['activities']
+            if item_index < len(activities):
+                return activities[item_index]
+        elif item_type == 'question' and section.get('questions'):
+            questions = section['questions']
+            if item_index < len(questions):
+                return questions[item_index]
+        
+        return None
+    
+    @staticmethod
+    def _advance_position(discussion_guide: Dict[str, Any], current_position: Dict[str, Any]) -> Tuple[Dict[str, Any], Optional[Dict[str, Any]], Dict[str, Any]]:
+        """
+        Advance the position to the next item in the discussion guide.
+        
+        Returns:
+            Tuple of (new_position, next_item, section_info)
+        """
+        sections = discussion_guide['sections']
+        section_index = current_position.get('section_index', 0)
+        item_index = current_position.get('item_index', 0)
+        item_type = current_position.get('item_type', 'activity')
+        
+        # Check if we're at the end
+        if section_index >= len(sections):
+            return current_position, None, {}
+        
+        current_section = sections[section_index]
+        
+        # Try to advance within the current section
+        if item_type == 'activity' and current_section.get('activities'):
+            activities = current_section['activities']
+            if item_index + 1 < len(activities):
+                # Next activity in same section
+                new_position = {
+                    'section_index': section_index,
+                    'item_index': item_index + 1,
+                    'item_type': 'activity'
+                }
+                return new_position, activities[item_index + 1], {'title': current_section.get('title', ''), 'type': current_section.get('type', '')}
+            else:
+                # Move to questions if available
+                if current_section.get('questions'):
+                    new_position = {
+                        'section_index': section_index,
+                        'item_index': 0,
+                        'item_type': 'question'
+                    }
+                    return new_position, current_section['questions'][0], {'title': current_section.get('title', ''), 'type': current_section.get('type', '')}
+        
+        elif item_type == 'question' and current_section.get('questions'):
+            questions = current_section['questions']
+            if item_index + 1 < len(questions):
+                # Next question in same section
+                new_position = {
+                    'section_index': section_index,
+                    'item_index': item_index + 1,
+                    'item_type': 'question'
+                }
+                return new_position, questions[item_index + 1], {'title': current_section.get('title', ''), 'type': current_section.get('type', '')}
+        
+        # Move to next section
+        next_section_index = section_index + 1
+        if next_section_index >= len(sections):
+            return current_position, None, {}  # End of guide
+        
+        next_section = sections[next_section_index]
+        
+        # Start with activities if available, otherwise questions
+        if next_section.get('activities'):
+            new_position = {
+                'section_index': next_section_index,
+                'item_index': 0,
+                'item_type': 'activity'
+            }
+            return new_position, next_section['activities'][0], {'title': next_section.get('title', ''), 'type': next_section.get('type', '')}
+        elif next_section.get('questions'):
+            new_position = {
+                'section_index': next_section_index,
+                'item_index': 0,
+                'item_type': 'question'
+            }
+            return new_position, next_section['questions'][0], {'title': next_section.get('title', ''), 'type': next_section.get('type', '')}
+        
+        # Section has no items, skip to next
+        return AIModeratorService._advance_position(discussion_guide, {
+            'section_index': next_section_index,
+            'item_index': 0,
+            'item_type': 'activity'
+        })
+    
+    @staticmethod
+    def _generate_moderator_response(focus_group_id: str, item: Dict[str, Any], section_info: Dict[str, Any], position: Dict[str, Any]) -> str:
+        """
+        Generate an appropriate moderator response for the current item.
+        
+        Args:
+            focus_group_id: The focus group ID
+            item: The current item (activity or question)
+            section_info: Information about the current section
+            position: Current position in the guide
+            
+        Returns:
+            Generated moderator response
+        """
+        try:
+            # Get previous messages for context
+            messages = FocusGroup.get_messages(focus_group_id)
+            recent_messages = messages[-10:] if messages else []  # Last 10 messages
+            
+            # Format context
+            context = {
+                'item_content': item.get('content', ''),
+                'item_type': item.get('type', 'unknown'),
+                'section_title': section_info.get('title', 'Unknown'),
+                'section_type': section_info.get('type', 'unknown'),
+                'recent_messages': AIModeratorService._format_messages_for_context(recent_messages),
+                'probes': item.get('probes', []) if item.get('probes') else [],
+                'time_limit': item.get('time_limit', 0) if item.get('time_limit') else 0
+            }
+            
+            # Load moderator prompt
+            prompt = load_prompt('ai-moderator-system', context)
+            
+            # Generate response
+            response = LLMService.generate_content(
+                prompt=prompt,
+                temperature=0.7
+            )
+            
+            return response.strip()
+            
+        except Exception as e:
+            # Fallback to item content if generation fails
+            return item.get('content', 'Let\'s continue with our discussion.')
+    
+    @staticmethod
+    def _format_messages_for_context(messages: List[Dict[str, Any]]) -> str:
+        """Format messages for use in moderator context."""
+        if not messages:
+            return "No previous messages."
+        
+        formatted = []
+        for msg in messages:
+            sender = msg.get('senderId', 'Unknown')
+            text = msg.get('text', '')
+            msg_type = msg.get('type', 'response')
+            
+            if msg_type == 'question':
+                formatted.append(f"MODERATOR: {text}")
+            elif msg_type == 'system':
+                formatted.append(f"SYSTEM: {text}")
+            else:
+                formatted.append(f"{sender}: {text}")
+        
+        return "\n".join(formatted)
+    
+    @staticmethod
+    def _handle_legacy_advance(focus_group_id: str, discussion_guide: str) -> Dict[str, Any]:
+        """Handle advancement for legacy markdown format guides."""
+        # For legacy format, we'll generate a generic moderator response
+        # This is a fallback for older discussion guides
+        try:
+            # Get recent messages for context
+            messages = FocusGroup.get_messages(focus_group_id)
+            recent_messages = messages[-5:] if messages else []
+            
+            # Create a simple context
+            context = {
+                'discussion_guide': discussion_guide,
+                'recent_messages': AIModeratorService._format_messages_for_context(recent_messages),
+                'legacy_mode': True
+            }
+            
+            # Try to load and use the moderator prompt
+            prompt = load_prompt('ai-moderator-system', context)
+            
+            response = LLMService.generate_content(
+                prompt=prompt,
+                temperature=0.7
+            )
+            
+            return {
+                "message": "Discussion advanced (legacy mode)",
+                "moderator_response": response.strip(),
+                "legacy_format": True
+            }
+            
+        except Exception as e:
+            return {
+                "message": "Discussion advanced (legacy mode)",
+                "moderator_response": "Let's continue with our discussion. What are your thoughts on the current topic?",
+                "legacy_format": True,
+                "error": str(e)
+            }
+    
+    @staticmethod
+    def end_session_with_concluding_statement(focus_group_id: str, reason: str = 'session_ended') -> Dict[str, Any]:
+        """
+        End a focus group session with a concluding moderator statement.
+        
+        Args:
+            focus_group_id: The focus group ID
+            reason: Reason for ending ('manual_stop', 'auto_complete', 'timeout', etc.)
+            
+        Returns:
+            Dictionary containing the concluding statement and session end confirmation
+        """
+        try:
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            # Generate concluding statement
+            concluding_message = AIModeratorService._generate_concluding_statement(
+                focus_group_id, reason
+            )
+            
+            # Save the concluding message
+            message_data = {
+                "text": concluding_message,
+                "type": "conclusion",
+                "senderId": "moderator"
+            }
+            
+            message_id = FocusGroup.add_message(focus_group_id, message_data)
+            
+            if not message_id:
+                print(f"Warning: Failed to save concluding message for focus group {focus_group_id}")
+            
+            # Update focus group status to completed
+            FocusGroup.update(focus_group_id, {
+                'status': 'completed'
+            })
+            
+            print(f"🎬 Session ended for focus group {focus_group_id} with reason: {reason}")
+            
+            return {
+                "message": "Session ended successfully",
+                "concluding_statement": concluding_message,
+                "reason": reason,
+                "focus_group_id": focus_group_id,
+                "message_id": message_id,
+                "status": "completed"
+            }
+            
+        except Exception as e:
+            return {"error": f"Error ending session: {str(e)}"}
+    
+    @staticmethod
+    def _generate_concluding_statement(focus_group_id: str, reason: str) -> str:
+        """
+        Generate an appropriate concluding statement for the session.
+        
+        Args:
+            focus_group_id: The focus group ID
+            reason: Reason for ending the session
+            
+        Returns:
+            Generated concluding statement
+        """
+        try:
+            # Get focus group details for context
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                return AIModeratorService._get_fallback_concluding_message(reason)
+            
+            # Get recent messages for context
+            messages = FocusGroup.get_messages(focus_group_id)
+            recent_messages = messages[-5:] if messages else []
+            
+            # Create context for LLM
+            context = {
+                'focus_group_name': focus_group.get('name', 'focus group'),
+                'focus_group_topic': focus_group.get('topic', 'discussion'),
+                'ending_reason': reason,
+                'recent_messages': AIModeratorService._format_messages_for_context(recent_messages),
+                'session_concluded': True
+            }
+            
+            # Try to generate with LLM using moderator prompt
+            prompt = load_prompt('ai-moderator-system', context)
+            
+            response = LLMService.generate_content(
+                prompt=prompt,
+                temperature=0.5  # Lower temperature for more consistent, professional responses
+            )
+            
+            return response.strip()
+            
+        except Exception as e:
+            print(f"Warning: Failed to generate concluding statement with LLM: {e}")
+            return AIModeratorService._get_fallback_concluding_message(reason)
+    
+    @staticmethod
+    def _get_fallback_concluding_message(reason: str) -> str:
+        """
+        Get a fallback concluding message when LLM generation fails.
+        
+        Args:
+            reason: Reason for ending the session
+            
+        Returns:
+            Appropriate fallback message
+        """
+        messages = {
+            'manual_stop': "The focus group session has now ended. Thank you for your participation and valuable insights.",
+            'auto_complete': "We have covered all topics in our discussion guide. Thank you everyone for your thoughtful participation in today's focus group.",
+            'timeout': "Our time for today's session has concluded. Thank you all for sharing your perspectives and contributing to this discussion.",
+            'session_ended': "The focus group session has now ended. Thank you for your participation."
+        }
+        
+        return messages.get(reason, messages['session_ended'])

backend/app/services/ai_persona_service.py

@@ -0,0 +1,648 @@
+"""
+AI Persona Generation Service using Google's Gemini model.
+This service handles the integration with the Gemini API to generate
+synthetic persona data based on a predefined prompt.
+"""
+
+import os
+import json
+import uuid
+from typing import Dict, Any, Optional, List
+from pydantic import BaseModel, ValidationError
+from datetime import datetime
+
+from .llm_service import LLMService, LLMServiceError
+from .customer_data_service import customer_data_service
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+
+
+
+
+
+class PersonaGenerationError(Exception):
+    """Exception raised for errors in the persona generation process."""
+    pass
+
+
+def _sanitize_persona_data_for_json(persona_data: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Sanitize persona data to make it JSON serializable.
+    
+    Args:
+        persona_data: The persona data dictionary that may contain non-serializable objects
+        
+    Returns:
+        A sanitized dictionary that can be JSON serialized
+    """
+    sanitized = {}
+    
+    for key, value in persona_data.items():
+        if isinstance(value, datetime):
+            # Convert datetime to ISO string
+            sanitized[key] = value.isoformat()
+        elif isinstance(value, dict):
+            # Recursively sanitize nested dictionaries
+            sanitized[key] = _sanitize_persona_data_for_json(value)
+        elif isinstance(value, list):
+            # Sanitize list items
+            sanitized[key] = [
+                _sanitize_persona_data_for_json(item) if isinstance(item, dict) 
+                else item.isoformat() if isinstance(item, datetime)
+                else item
+                for item in value
+            ]
+        else:
+            # Keep other values as-is
+            sanitized[key] = value
+    
+    return sanitized
+
+
+def generate_basic_personas(
+    audience_brief: str,
+    research_objective: Optional[str] = None,
+    count: int = 5,
+    temperature: float = 0.8,
+    customer_data_session_id: Optional[str] = None
+) -> List[Dict[str, Any]]:
+    """
+    Generate basic profiles for multiple personas based on a research brief.
+    
+    Args:
+        audience_brief: The audience brief to guide persona generation
+        research_objective: Optional research objective to focus persona goals and scenarios
+        count: Number of basic personas to generate
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        customer_data_session_id: Optional session ID for customer data context
+        
+    Returns:
+        A list of dictionaries containing basic persona data
+        
+    Raises:
+        PersonaGenerationError: If there's an issue with the AI generation or JSON parsing
+    """
+    try:
+        # Load customer data context if session ID provided
+        customer_data_context = ''
+        if customer_data_session_id:
+            customer_data_content = customer_data_service.get_parsed_markdown_content(customer_data_session_id)
+            if customer_data_content:
+                customer_data_context = f"The following customer data was uploaded and should be used to inform persona creation:\n\n{customer_data_content}"
+            else:
+                customer_data_context = "No customer data available for this session."
+        else:
+            customer_data_context = "No customer data provided."
+        
+        # Load and format the prompt with the audience brief and count
+        try:
+            final_prompt = load_prompt('persona-basic-generation', {
+                'audience_brief': audience_brief,
+                'research_objective': research_objective or '',
+                'count': count,
+                'customer_data_context': customer_data_context
+            })
+        except PromptLoaderError as e:
+            raise PersonaGenerationError(f"Error loading prompt: {str(e)}")
+        
+        # Add additional safeguards for JSON parsing
+        try:
+            # Load system prompt and generate raw content
+            try:
+                system_prompt = load_prompt('persona-system')
+            except PromptLoaderError as e:
+                raise PersonaGenerationError(f"Error loading system prompt: {str(e)}")
+            
+            raw_response = LLMService.generate_content(
+                prompt=final_prompt,
+                temperature=temperature,
+                system_prompt=system_prompt
+            )
+            
+            # Try to clean up the response for proper JSON parsing
+            clean_response = raw_response
+            
+            # Remove markdown code blocks if present
+            if clean_response.startswith("```json"):
+                clean_response = clean_response.strip("```json").strip("```").strip()
+            elif clean_response.startswith("```"):
+                clean_response = clean_response.strip("```").strip()
+            
+            # Try to find the JSON array in the response if there's extra text
+            if not clean_response.startswith("["):
+                # Look for the opening bracket
+                start_idx = clean_response.find("[")
+                if start_idx != -1:
+                    # Find the matching closing bracket
+                    end_idx = clean_response.rfind("]")
+                    if end_idx != -1 and end_idx > start_idx:
+                        clean_response = clean_response[start_idx:end_idx+1]
+            
+            # Parse the JSON manually
+            try:
+                print(f"Attempting to parse JSON array: {clean_response[:100]}...")
+                personas_array = json.loads(clean_response)
+                
+                # Verify it's an array
+                if not isinstance(personas_array, list):
+                    raise PersonaGenerationError(f"Expected an array of personas but got {type(personas_array)}")
+                    
+            except json.JSONDecodeError as e:
+                raise PersonaGenerationError(f"Failed to parse JSON response: {str(e)}. Raw response: {clean_response[:200]}...")
+            
+        except LLMServiceError as e:
+            raise PersonaGenerationError(f"Error from LLM service: {str(e)}")
+        
+        # Validate we got an array with the right count
+        if not isinstance(personas_array, list):
+            raise PersonaGenerationError(f"Expected an array of personas but got {type(personas_array)}")
+        
+        # Check if we got at least one persona
+        if len(personas_array) == 0:
+            raise PersonaGenerationError("No personas were generated")
+        
+        # If we got fewer personas than requested, log a warning but continue
+        if len(personas_array) < count:
+            print(f"Warning: Requested {count} personas but only got {len(personas_array)}")
+        
+        # Basic validation of each persona
+        required_fields = ["name", "age", "gender", "occupation", "personality"]
+        for i, persona in enumerate(personas_array):
+            missing_fields = [field for field in required_fields if field not in persona]
+            if missing_fields:
+                raise PersonaGenerationError(
+                    f"Persona {i+1} is missing required fields: {', '.join(missing_fields)}"
+                )
+        
+        return personas_array
+        
+    except Exception as e:
+        if isinstance(e, PersonaGenerationError):
+            raise
+        raise PersonaGenerationError(f"Error generating basic personas: {str(e)}")
+
+
+def generate_persona(
+    prompt_customization: Optional[str] = None,
+    basic_persona: Optional[Dict[str, Any]] = None,
+    temperature: float = 0.7,
+    customer_data_session_id: Optional[str] = None
+) -> Dict[str, Any]:
+    """
+    Generate a synthetic persona using the Gemini model.
+    
+    Args:
+        prompt_customization: Optional string to customize the generation
+        basic_persona: Optional dictionary containing basic persona data to start with
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        customer_data_session_id: Optional session ID for customer data context
+        
+    Returns:
+        A dictionary containing the generated persona data
+        
+    Raises:
+        PersonaGenerationError: If there's an issue with the AI generation or JSON parsing
+    """
+    try:
+        # Load customer data context if session ID provided
+        customer_data_context = ''
+        if customer_data_session_id:
+            customer_data_content = customer_data_service.get_parsed_markdown_content(customer_data_session_id)
+            if customer_data_content:
+                customer_data_context = f"The following customer data was uploaded and should be used to inform persona creation:\n\n{customer_data_content}"
+            else:
+                customer_data_context = "No customer data available for this session."
+        else:
+            customer_data_context = "No customer data provided."
+        
+        # Load the base prompt
+        try:
+            final_prompt = load_prompt('persona-detailed-generation', {
+                'customer_data_context': customer_data_context
+            })
+        except PromptLoaderError as e:
+            raise PersonaGenerationError(f"Error loading prompt: {str(e)}")
+        
+        # Add customization if provided
+        if prompt_customization:
+            final_prompt = f"{final_prompt}\n\nAdditional customization: {prompt_customization}"
+        
+        # Add basic persona data if provided
+        if basic_persona:
+            # Create a prompt section with the basic persona data
+            basic_data_str = "\nUse this basic profile as a starting point:\n"
+            basic_data_str += json.dumps(basic_persona, indent=2)
+            basic_data_str += "\n\nMaintain the demographic information above while expanding the persona with goals, frustrations, motivations, etc."
+            
+            final_prompt = f"{final_prompt}\n{basic_data_str}"
+        
+        try:
+            # Load system prompt and generate structured response
+            try:
+                system_prompt = load_prompt('persona-system')
+            except PromptLoaderError as e:
+                raise PersonaGenerationError(f"Error loading system prompt: {str(e)}")
+            
+            persona_data = LLMService.generate_structured_response(
+                prompt=final_prompt,
+                temperature=temperature,
+                system_prompt=system_prompt
+            )
+            
+        except LLMServiceError as e:
+            raise PersonaGenerationError(f"Error from LLM service: {str(e)}")
+        
+        # Validate the required fields
+        required_fields = ["name", "age", "gender", "occupation", "location", "techSavviness", "personality"]
+        missing_fields = [field for field in required_fields if field not in persona_data]
+        
+        if missing_fields:
+            raise PersonaGenerationError(f"Generated persona is missing required fields: {', '.join(missing_fields)}")
+        
+        # Generate ID if missing
+        if "id" not in persona_data:
+            persona_data["id"] = f"generated-{uuid.uuid4()}"
+            
+        return persona_data
+        
+    except Exception as e:
+        if isinstance(e, PersonaGenerationError):
+            raise
+        raise PersonaGenerationError(f"Error generating persona: {str(e)}")
+
+
+def generate_persona_summary(
+    persona_data: Dict[str, Any],
+    temperature: float = 0.7
+) -> Dict[str, Any]:
+    """
+    Generate a concise summary of a persona for display on persona cards.
+    
+    Args:
+        persona_data: The complete persona data dictionary
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        
+    Returns:
+        A dictionary containing aiSynthesizedBio, qualitativeAttributes, and topPersonalityTraits
+        
+    Raises:
+        PersonaGenerationError: If there's an issue with the AI generation or JSON parsing
+    """
+    try:
+        # Sanitize persona data for JSON serialization
+        sanitized_persona_data = _sanitize_persona_data_for_json(persona_data)
+        
+        # Load and format the prompt with the persona data
+        try:
+            final_prompt = load_prompt('persona-summary-generation', {
+                'persona_data': json.dumps(sanitized_persona_data, indent=2)
+            })
+        except PromptLoaderError as e:
+            raise PersonaGenerationError(f"Error loading summary prompt: {str(e)}")
+        
+        try:
+            # Load system prompt and generate structured response
+            try:
+                system_prompt = load_prompt('persona-system')
+            except PromptLoaderError as e:
+                raise PersonaGenerationError(f"Error loading system prompt: {str(e)}")
+            
+            raw_response = LLMService.generate_content(
+                prompt=final_prompt,
+                temperature=temperature,
+                system_prompt=system_prompt
+            )
+            
+            # Clean up the response for proper JSON parsing
+            clean_response = raw_response.strip()
+            
+            # Remove markdown code blocks if present
+            if clean_response.startswith("```json"):
+                clean_response = clean_response.strip("```json").strip("```").strip()
+            elif clean_response.startswith("```"):
+                clean_response = clean_response.strip("```").strip()
+            
+            # Try to find the JSON object in the response if there's extra text
+            if not clean_response.startswith("{"):
+                # Look for the opening brace
+                start_idx = clean_response.find("{")
+                if start_idx != -1:
+                    # Find the matching closing brace
+                    end_idx = clean_response.rfind("}")
+                    if end_idx != -1 and end_idx > start_idx:
+                        clean_response = clean_response[start_idx:end_idx+1]
+            
+            # Parse the JSON manually
+            try:
+                print(f"Attempting to parse summary JSON: {clean_response[:100]}...")
+                summary_data = json.loads(clean_response)
+                
+                # Verify it's a dictionary with required fields
+                if not isinstance(summary_data, dict):
+                    raise PersonaGenerationError(f"Expected a summary object but got {type(summary_data)}")
+                
+                required_fields = ["aiSynthesizedBio", "qualitativeAttributes", "topPersonalityTraits"]
+                missing_fields = [field for field in required_fields if field not in summary_data]
+                
+                if missing_fields:
+                    raise PersonaGenerationError(f"Summary is missing required fields: {', '.join(missing_fields)}")
+                
+                # Validate field types
+                if not isinstance(summary_data["aiSynthesizedBio"], str):
+                    raise PersonaGenerationError("aiSynthesizedBio must be a string")
+                if not isinstance(summary_data["qualitativeAttributes"], list):
+                    raise PersonaGenerationError("qualitativeAttributes must be an array")
+                if not isinstance(summary_data["topPersonalityTraits"], list):
+                    raise PersonaGenerationError("topPersonalityTraits must be an array")
+                
+                return summary_data
+                    
+            except json.JSONDecodeError as e:
+                raise PersonaGenerationError(f"Failed to parse summary JSON response: {str(e)}. Raw response: {clean_response[:200]}...")
+            
+        except LLMServiceError as e:
+            raise PersonaGenerationError(f"Error from LLM service: {str(e)}")
+        
+    except Exception as e:
+        if isinstance(e, PersonaGenerationError):
+            raise
+        raise PersonaGenerationError(f"Error generating persona summary: {str(e)}")
+
+
+def generate_persona_download_summary(
+    persona_data: Dict[str, Any],
+    temperature: float = 0.7
+) -> str:
+    """
+    Generate a comprehensive markdown summary of a persona for download/client review.
+    
+    Args:
+        persona_data: The complete persona data dictionary
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        
+    Returns:
+        A string containing the markdown-formatted persona summary
+        
+    Raises:
+        PersonaGenerationError: If there's an issue with the AI generation
+    """
+    try:
+        # Sanitize persona data for JSON serialization
+        sanitized_persona_data = _sanitize_persona_data_for_json(persona_data)
+        
+        # Load and format the prompt with the persona data
+        try:
+            final_prompt = load_prompt('persona-download-summary', {
+                'persona_data': json.dumps(sanitized_persona_data, indent=2)
+            })
+        except PromptLoaderError as e:
+            raise PersonaGenerationError(f"Error loading download summary prompt: {str(e)}")
+        
+        try:
+            # Load system prompt and generate markdown response
+            try:
+                system_prompt = load_prompt('persona-system')
+            except PromptLoaderError as e:
+                raise PersonaGenerationError(f"Error loading system prompt: {str(e)}")
+            
+            # Generate the markdown content directly
+            markdown_response = LLMService.generate_content(
+                prompt=final_prompt,
+                temperature=temperature,
+                system_prompt=system_prompt
+            )
+            
+            # Clean up the response if needed
+            clean_response = markdown_response.strip()
+            
+            # Remove markdown code blocks if present
+            if clean_response.startswith("```markdown"):
+                clean_response = clean_response.strip("```markdown").strip("```").strip()
+            elif clean_response.startswith("```"):
+                clean_response = clean_response.strip("```").strip()
+            
+            return clean_response
+            
+        except LLMServiceError as e:
+            raise PersonaGenerationError(f"Error from LLM service: {str(e)}")
+        
+    except Exception as e:
+        if isinstance(e, PersonaGenerationError):
+            raise
+        raise PersonaGenerationError(f"Error generating persona download summary: {str(e)}")
+
+
+def customize_persona_prompt(
+    age_range: Optional[str] = None,
+    gender: Optional[str] = None,
+    occupation_type: Optional[str] = None,
+    education_level: Optional[str] = None,
+    location_type: Optional[str] = None,
+    personality_traits: Optional[str] = None,
+    interests: Optional[str] = None,
+    audience_brief: Optional[str] = None,
+    research_objective: Optional[str] = None
+) -> Optional[str]:
+    """
+    Create a customized prompt for more specific persona generation.
+    
+    Args:
+        age_range: Age range for the persona
+        gender: Gender of the persona
+        occupation_type: Type of occupation
+        education_level: Level of education
+        location_type: Geographic location
+        personality_traits: Personality characteristics
+        interests: Personal interests and hobbies
+        audience_brief: Full audience brief providing context for persona generation
+        research_objective: Research objective to focus persona goals, frustrations, and scenarios
+        
+    Returns:
+        A string with customization instructions or None if no customizations provided
+    """
+    customizations = []
+    
+    # If an audience brief is provided, use it first as it provides the most context
+    if audience_brief or research_objective:
+        prompt = ""
+        if audience_brief:
+            prompt += f"""
+Audience Brief:
+{audience_brief}
+"""
+        if research_objective:
+            prompt += f"""
+Research Objective:
+{research_objective}
+"""
+        
+        prompt += "\nBased on the above context, create a persona that would be relevant to this research."
+        
+        if research_objective:
+            prompt += f"""
+
+CRITICAL RESEARCH ALIGNMENT: This persona MUST be designed around the research objective: '{research_objective}'. 
+
+LIFE SCENARIOS REQUIREMENTS:
+- At least 3 out of 5 scenarios MUST show this persona directly encountering, using, deciding about, or being impacted by aspects of: {research_objective}
+- Each research-aligned scenario must be a specific, realistic situation showing their authentic relationship with this topic
+- Show varied contexts: work situations, personal decisions, social interactions, consumer experiences - all demonstrating how '{research_objective}' appears in their real life
+- Scenarios should reveal the persona's thoughts, feelings, and behaviors when dealing with this research topic
+- Include both positive and challenging experiences related to the research focus
+- Make scenarios concrete and specific to this research objective, not generic situations"""
+        
+        if customizations:
+            prompt += f"\nAdditionally, ensure the persona meets these specific requirements: {'; '.join(customizations)}"
+        
+        return prompt
+    
+    # Otherwise, use the individual parameters
+    if age_range:
+        customizations.append(f"Age range: {age_range}")
+    if gender:
+        customizations.append(f"Gender: {gender}")
+    if occupation_type:
+        customizations.append(f"Occupation type: {occupation_type}")
+    if education_level:
+        customizations.append(f"Education level: {education_level}")
+    if location_type:
+        customizations.append(f"Location: {location_type}")
+    if personality_traits:
+        customizations.append(f"Personality traits: {personality_traits}")
+    if interests:
+        customizations.append(f"Interests: {interests}")
+        
+    if not customizations:
+        return None
+        
+    return "Create a persona with these characteristics: " + "; ".join(customizations)
+
+
+def enhance_audience_brief(
+    audience_brief: str,
+    research_objective: str,
+    temperature: float = 0.7
+) -> Dict[str, List[str]]:
+    """
+    Generate suggestions to improve both audience brief and research objective for better persona generation.
+    
+    Args:
+        audience_brief: The audience brief to analyze and improve
+        research_objective: The research objective to analyze and improve
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        
+    Returns:
+        A dictionary with separate suggestion arrays for 'audience_brief' and 'research_objective'
+        
+    Raises:
+        PersonaGenerationError: If there's an issue with the AI generation or JSON parsing
+    """
+    try:
+        # Load and format the prompt with both fields
+        try:
+            final_prompt = load_prompt('audience-brief-enhancement', {
+                'audience_brief': audience_brief,
+                'research_objective': research_objective
+            })
+        except PromptLoaderError as e:
+            raise PersonaGenerationError(f"Error loading enhancement prompt: {str(e)}")
+        
+        # Generate suggestions using the LLM service
+        try:
+            raw_response = LLMService.generate_content(
+                prompt=final_prompt,
+                temperature=temperature
+            )
+            
+            # DEBUG: Log the raw response from the LLM
+            print(f"DEBUG: Raw LLM response: {raw_response[:500]}...")
+            
+            # Clean up the response for proper JSON parsing
+            clean_response = raw_response.strip()
+            
+            # DEBUG: Log the cleaned response
+            print(f"DEBUG: Cleaned response: {clean_response[:500]}...")
+            
+            # Remove markdown code blocks if present
+            if clean_response.startswith("```json"):
+                clean_response = clean_response.strip("```json").strip("```").strip()
+            elif clean_response.startswith("```"):
+                clean_response = clean_response.strip("```").strip()
+            
+            # Try to find the JSON object in the response if there's extra text
+            if not clean_response.startswith("{"):
+                # Look for the opening brace
+                start_idx = clean_response.find("{")
+                if start_idx != -1:
+                    # Find the matching closing brace
+                    end_idx = clean_response.rfind("}")
+                    if end_idx != -1 and end_idx > start_idx:
+                        clean_response = clean_response[start_idx:end_idx+1]
+            
+            # Parse the JSON response
+            try:
+                suggestions_object = json.loads(clean_response)
+                
+                # DEBUG: Log the entire parsed response for troubleshooting
+                print(f"DEBUG: Parsed suggestions object: {json.dumps(suggestions_object, indent=2)}")
+                
+                # Verify it's an object
+                if not isinstance(suggestions_object, dict):
+                    raise PersonaGenerationError(f"Expected a JSON object with suggestions but got {type(suggestions_object)}")
+                
+                # Verify required keys exist
+                if 'audience_brief' not in suggestions_object or 'research_objective' not in suggestions_object:
+                    raise PersonaGenerationError("Response must contain both 'audience_brief' and 'research_objective' keys")
+                
+                # Verify both values are arrays
+                if not isinstance(suggestions_object['audience_brief'], list):
+                    raise PersonaGenerationError(f"audience_brief must be an array but got {type(suggestions_object['audience_brief'])}")
+                if not isinstance(suggestions_object['research_objective'], list):
+                    raise PersonaGenerationError(f"research_objective must be an array but got {type(suggestions_object['research_objective'])}")
+                
+                # Verify all items in both arrays are strings - with detailed debugging and auto-fixing
+                for i, suggestion in enumerate(suggestions_object['audience_brief']):
+                    if not isinstance(suggestion, str):
+                        print(f"DEBUG: audience_brief[{i}] type: {type(suggestion)}, value: {suggestion}")
+                        # Try to convert to string if it's a dict with a text field, otherwise stringify
+                        if isinstance(suggestion, dict) and 'text' in suggestion:
+                            suggestions_object['audience_brief'][i] = suggestion['text']
+                            print(f"DEBUG: Fixed audience_brief[{i}] by extracting 'text' field: {suggestion['text']}")
+                        elif isinstance(suggestion, dict):
+                            # Convert dict to string representation
+                            suggestions_object['audience_brief'][i] = str(suggestion)
+                            print(f"DEBUG: Fixed audience_brief[{i}] by converting dict to string")
+                        else:
+                            suggestions_object['audience_brief'][i] = str(suggestion)
+                            print(f"DEBUG: Fixed audience_brief[{i}] by converting to string")
+                        
+                for i, suggestion in enumerate(suggestions_object['research_objective']):
+                    if not isinstance(suggestion, str):
+                        print(f"DEBUG: research_objective[{i}] type: {type(suggestion)}, value: {suggestion}")
+                        # Try to convert to string if it's a dict with a text field, otherwise stringify
+                        if isinstance(suggestion, dict) and 'text' in suggestion:
+                            suggestions_object['research_objective'][i] = suggestion['text']
+                            print(f"DEBUG: Fixed research_objective[{i}] by extracting 'text' field: {suggestion['text']}")
+                        elif isinstance(suggestion, dict):
+                            # Convert dict to string representation
+                            suggestions_object['research_objective'][i] = str(suggestion)
+                            print(f"DEBUG: Fixed research_objective[{i}] by converting dict to string")
+                        else:
+                            suggestions_object['research_objective'][i] = str(suggestion)
+                            print(f"DEBUG: Fixed research_objective[{i}] by converting to string")
+                        
+            except json.JSONDecodeError as e:
+                raise PersonaGenerationError(f"Failed to parse JSON response: {str(e)}. Raw response: {clean_response[:200]}...")
+            
+        except LLMServiceError as e:
+            raise PersonaGenerationError(f"Error from LLM service: {str(e)}")
+        
+        # Validate we got at least one suggestion in each field
+        if len(suggestions_object['audience_brief']) == 0 and len(suggestions_object['research_objective']) == 0:
+            raise PersonaGenerationError("No suggestions were generated for either field")
+        
+        return suggestions_object
+        
+    except Exception as e:
+        if isinstance(e, PersonaGenerationError):
+            raise
+        raise PersonaGenerationError(f"Error enhancing audience brief: {str(e)}")

backend/app/services/conversation_context_service.py

@@ -0,0 +1,730 @@
+"""
+Conversation Context Service
+Aggregates and formats all context needed for LLM-based conversation decisions.
+Also handles multimodal conversation context building with visual assets.
+"""
+
+from typing import Dict, List, Any, Optional
+from datetime import datetime, timedelta
+import json
+import os
+from collections import defaultdict, Counter
+
+from app.models.focus_group import FocusGroup
+from app.models.persona import Persona
+
+
+class ConversationContextService:
+    """Service for aggregating conversation context for LLM decision making."""
+    
+    @staticmethod
+    def get_full_context(focus_group_id: str) -> Dict[str, Any]:
+        """
+        Get complete conversation context for LLM decision making.
+        
+        Args:
+            focus_group_id: The focus group ID
+            
+        Returns:
+            Dictionary containing all context for LLM decision making
+        """
+        try:
+            # Get focus group data
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                raise ValueError(f"Focus group {focus_group_id} not found")
+            
+            # Get all participants
+            participants = ConversationContextService._get_participants_context(focus_group)
+            
+            # Get conversation history
+            messages = FocusGroup.get_messages(focus_group_id)
+            conversation_history = ConversationContextService._format_conversation_history(messages)
+            
+            # Get conversation analytics
+            analytics = ConversationContextService._analyze_conversation(messages, participants)
+            
+            # Get discussion guide context
+            discussion_guide_context = ConversationContextService._get_discussion_guide_context(focus_group)
+            
+            # Calculate elapsed time
+            created_at = focus_group.get('created_at')
+            if created_at:
+                elapsed_minutes = (datetime.utcnow() - created_at).total_seconds() / 60
+            else:
+                elapsed_minutes = 0
+            
+            return {
+                'focus_group_topic': focus_group.get('topic', 'Unknown'),
+                'focus_group_duration': focus_group.get('duration', 60),
+                'current_time': round(elapsed_minutes, 1),
+                'discussion_guide_context': discussion_guide_context,
+                'current_section': ConversationContextService._get_current_section(focus_group),
+                'participants_context': participants,
+                'conversation_history': conversation_history,
+                'conversation_analytics': analytics
+            }
+            
+        except Exception as e:
+            raise Exception(f"Error getting conversation context: {str(e)}")
+    
+    @staticmethod
+    def _get_participants_context(focus_group: Dict[str, Any]) -> List[Dict[str, Any]]:
+        """Get formatted participant context with OCEAN traits and participation stats."""
+        participants = []
+        participant_ids = focus_group.get('participants', [])
+        
+        for participant_id in participant_ids:
+            persona = Persona.find_by_id(participant_id)
+            if persona:
+                participant_context = {
+                    'id': participant_id,
+                    'name': persona.get('name', 'Unknown'),
+                    'demographics': {
+                        'age': persona.get('age', 'Unknown'),
+                        'gender': persona.get('gender', 'Unknown'),
+                        'occupation': persona.get('occupation', 'Unknown'),
+                        'location': persona.get('location', 'Unknown'),
+                        'education': persona.get('education', 'Unknown')
+                    },
+                    'personality': {
+                        'description': persona.get('personality', 'No description'),
+                        'ocean_traits': persona.get('oceanTraits', {}),
+                        'goals': persona.get('goals', []),
+                        'frustrations': persona.get('frustrations', []),
+                        'motivations': persona.get('motivations', [])
+                    },
+                    'interests': persona.get('interests', ''),
+                    'background': {
+                        'think_feel_do': persona.get('thinkFeelDo', {}),
+                        'scenarios': persona.get('scenarios', [])
+                    }
+                }
+                participants.append(participant_context)
+        
+        return participants
+    
+    @staticmethod
+    def _format_conversation_history(messages: List[Dict[str, Any]], limit: int = 20) -> List[Dict[str, Any]]:
+        """Format recent conversation history for LLM consumption."""
+        if not messages:
+            return []
+        
+        # Get recent messages
+        recent_messages = messages[-limit:] if len(messages) > limit else messages
+        
+        formatted_messages = []
+        for msg in recent_messages:
+            formatted_msg = {
+                'id': msg.get('_id', msg.get('id', 'unknown')),
+                'sender_id': msg.get('senderId', 'unknown'),
+                'sender_type': 'moderator' if msg.get('senderId') == 'moderator' else 'participant',
+                'message_type': msg.get('type', 'response'),
+                'content': msg.get('text', ''),
+                'timestamp': msg.get('created_at', ''),
+                'highlighted': msg.get('highlighted', False)
+            }
+            formatted_messages.append(formatted_msg)
+        
+        return formatted_messages
+    
+    @staticmethod
+    def _analyze_conversation(messages: List[Dict[str, Any]], participants: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Analyze conversation for participation patterns and sentiment."""
+        if not messages:
+            return {
+                'total_messages': 0,
+                'participant_stats': {},
+                'recent_activity': {},
+                'sentiment_analysis': {},
+                'topic_emergence': []
+            }
+        
+        # Participation statistics
+        participant_stats = defaultdict(lambda: {
+            'total_messages': 0,
+            'recent_messages': 0,
+            'last_message_index': -1,
+            'avg_message_length': 0,
+            'participation_percentage': 0
+        })
+        
+        # Count messages by participant
+        total_participant_messages = 0
+        message_lengths = defaultdict(list)
+        
+        for i, msg in enumerate(messages):
+            sender_id = msg.get('senderId')
+            if sender_id != 'moderator':
+                participant_stats[sender_id]['total_messages'] += 1
+                participant_stats[sender_id]['last_message_index'] = i
+                
+                message_length = len(msg.get('text', ''))
+                message_lengths[sender_id].append(message_length)
+                total_participant_messages += 1
+                
+                # Count recent messages (last 10)
+                if i >= len(messages) - 10:
+                    participant_stats[sender_id]['recent_messages'] += 1
+        
+        # Calculate averages and percentages
+        for participant_id, stats in participant_stats.items():
+            if message_lengths[participant_id]:
+                stats['avg_message_length'] = sum(message_lengths[participant_id]) / len(message_lengths[participant_id])
+            
+            if total_participant_messages > 0:
+                stats['participation_percentage'] = (stats['total_messages'] / total_participant_messages) * 100
+        
+        # Recent activity analysis
+        recent_messages = messages[-10:] if len(messages) > 10 else messages
+        recent_activity = {
+            'last_speaker': recent_messages[-1].get('senderId', 'unknown') if recent_messages else 'none',
+            'last_message_type': recent_messages[-1].get('type', 'unknown') if recent_messages else 'none',
+            'messages_in_last_10': len([m for m in recent_messages if m.get('senderId') != 'moderator']),
+            'unique_speakers_in_last_10': len(set([m.get('senderId') for m in recent_messages if m.get('senderId') != 'moderator']))
+        }
+        
+        # Basic sentiment analysis (simple keyword-based for now)
+        sentiment_analysis = ConversationContextService._analyze_sentiment(recent_messages)
+        
+        # Topic emergence detection
+        topic_emergence = ConversationContextService._detect_emerging_topics(recent_messages)
+        
+        return {
+            'total_messages': len(messages),
+            'participant_stats': dict(participant_stats),
+            'recent_activity': recent_activity,
+            'sentiment_analysis': sentiment_analysis,
+            'topic_emergence': topic_emergence
+        }
+    
+    @staticmethod
+    def _analyze_sentiment(messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Basic sentiment analysis of recent messages."""
+        if not messages:
+            return {'overall_sentiment': 'neutral', 'agreement_level': 0.5, 'energy_level': 0.5}
+        
+        # Simple keyword-based sentiment analysis
+        positive_words = ['good', 'great', 'excellent', 'love', 'like', 'amazing', 'wonderful', 'fantastic', 'agree', 'yes', 'definitely', 'absolutely']
+        negative_words = ['bad', 'terrible', 'hate', 'dislike', 'awful', 'horrible', 'disagree', 'no', 'never', 'wrong', 'problem', 'issue']
+        agreement_words = ['agree', 'yes', 'exactly', 'definitely', 'absolutely', 'same', 'similar', 'too', 'also']
+        disagreement_words = ['disagree', 'no', 'but', 'however', 'different', 'opposite', 'wrong', 'not']
+        
+        positive_count = 0
+        negative_count = 0
+        agreement_count = 0
+        disagreement_count = 0
+        total_words = 0
+        
+        for msg in messages:
+            text = msg.get('text', '').lower()
+            words = text.split()
+            total_words += len(words)
+            
+            for word in words:
+                if word in positive_words:
+                    positive_count += 1
+                elif word in negative_words:
+                    negative_count += 1
+                
+                if word in agreement_words:
+                    agreement_count += 1
+                elif word in disagreement_words:
+                    disagreement_count += 1
+        
+        # Calculate sentiment scores
+        if total_words > 0:
+            positive_ratio = positive_count / total_words
+            negative_ratio = negative_count / total_words
+            agreement_ratio = agreement_count / total_words
+            disagreement_ratio = disagreement_count / total_words
+        else:
+            positive_ratio = negative_ratio = agreement_ratio = disagreement_ratio = 0
+        
+        # Determine overall sentiment
+        if positive_ratio > negative_ratio * 1.5:
+            overall_sentiment = 'positive'
+        elif negative_ratio > positive_ratio * 1.5:
+            overall_sentiment = 'negative'
+        else:
+            overall_sentiment = 'neutral'
+        
+        # Calculate agreement level
+        if agreement_ratio + disagreement_ratio > 0:
+            agreement_level = agreement_ratio / (agreement_ratio + disagreement_ratio)
+        else:
+            agreement_level = 0.5
+        
+        return {
+            'overall_sentiment': overall_sentiment,
+            'agreement_level': agreement_level,
+            'energy_level': min(1.0, (positive_ratio + negative_ratio) * 10),  # Rough energy estimate
+            'sentiment_variance': abs(positive_ratio - negative_ratio)
+        }
+    
+    @staticmethod
+    def _detect_emerging_topics(messages: List[Dict[str, Any]]) -> List[str]:
+        """Detect emerging topics from recent conversation."""
+        if not messages:
+            return []
+        
+        # Simple topic detection based on word frequency
+        word_freq = Counter()
+        
+        # Common words to ignore
+        stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could', 'should', 'may', 'might', 'can', 'this', 'that', 'these', 'those', 'i', 'you', 'he', 'she', 'it', 'we', 'they', 'me', 'him', 'her', 'us', 'them', 'my', 'your', 'his', 'her', 'its', 'our', 'their'}
+        
+        for msg in messages:
+            text = msg.get('text', '').lower()
+            words = [word.strip('.,!?;:"()[]{}') for word in text.split()]
+            
+            for word in words:
+                if len(word) > 3 and word not in stop_words:
+                    word_freq[word] += 1
+        
+        # Get most frequent words as potential topics
+        emerging_topics = [word for word, count in word_freq.most_common(10) if count > 1]
+        
+        return emerging_topics
+    
+    @staticmethod
+    def _get_discussion_guide_context(focus_group: Dict[str, Any]) -> str:
+        """Get full detailed discussion guide context for the LLM."""
+        discussion_guide = focus_group.get('discussionGuide', '')
+        
+        if isinstance(discussion_guide, str):
+            return discussion_guide
+        elif isinstance(discussion_guide, dict):
+            # For structured discussion guides, provide complete detail for LLM decision making
+            title = discussion_guide.get('title', 'Unknown')
+            sections = discussion_guide.get('sections', [])
+            
+            context = f"DISCUSSION GUIDE: {title}\n"
+            context += f"Total Duration: {discussion_guide.get('total_duration', 'Unknown')} minutes\n\n"
+            
+            for i, section in enumerate(sections):
+                section_id = section.get('id', f'section-{i}')
+                section_title = section.get('title', 'Unknown Section')
+                section_duration = section.get('duration', 0)
+                section_type = section.get('type', 'unknown')
+                
+                context += f"SECTION {i+1}: {section_title} [ID: {section_id}]\n"
+                context += f"- Type: {section_type}\n"
+                context += f"- Duration: {section_duration} minutes\n"
+                
+                # Add section content if available
+                if section.get('content'):
+                    context += f"- Overview: {section['content']}\n"
+                
+                # Add detailed activities
+                activities = section.get('activities', [])
+                if activities:
+                    context += f"- Activities ({len(activities)}):\n"
+                    for j, activity in enumerate(activities):
+                        activity_id = activity.get('id', f'activity-{j}')
+                        activity_content = activity.get('content', 'No content')
+                        activity_time = activity.get('time_limit', 'No limit')
+                        context += f"  • [ID: {activity_id}] {activity_content} (Time: {activity_time} min)\n"
+                        
+                        # Add probes if available
+                        if activity.get('probes'):
+                            probes = activity['probes']
+                            if isinstance(probes, list) and probes:
+                                # Handle both string and dict probes
+                                probe_strings = []
+                                for probe in probes:
+                                    if isinstance(probe, str):
+                                        probe_strings.append(probe)
+                                    elif isinstance(probe, dict):
+                                        # Extract content from probe dict
+                                        probe_strings.append(probe.get('content', str(probe)))
+                                    else:
+                                        probe_strings.append(str(probe))
+                                context += f"    Probes: {'; '.join(probe_strings)}\n"
+                
+                # Add detailed questions
+                questions = section.get('questions', [])
+                if questions:
+                    context += f"- Questions ({len(questions)}):\n"
+                    for j, question in enumerate(questions):
+                        question_id = question.get('id', f'question-{j}')
+                        question_content = question.get('content', 'No content')
+                        question_time = question.get('time_limit', 'No limit')
+                        context += f"  • [ID: {question_id}] {question_content} (Time: {question_time} min)\n"
+                        
+                        # Add probes if available
+                        if question.get('probes'):
+                            probes = question['probes']
+                            if isinstance(probes, list) and probes:
+                                # Handle both string and dict probes
+                                probe_strings = []
+                                for probe in probes:
+                                    if isinstance(probe, str):
+                                        probe_strings.append(probe)
+                                    elif isinstance(probe, dict):
+                                        # Extract content from probe dict
+                                        probe_strings.append(probe.get('content', str(probe)))
+                                    else:
+                                        probe_strings.append(str(probe))
+                                context += f"    Probes: {'; '.join(probe_strings)}\n"
+                
+                # Add subsections if available
+                subsections = section.get('subsections', [])
+                if subsections:
+                    context += f"- Subsections ({len(subsections)}):\n"
+                    for subsection in subsections:
+                        subsection_id = subsection.get('id', 'unknown')
+                        subsection_title = subsection.get('title', 'Unknown')
+                        subsection_duration = subsection.get('duration', 0)
+                        context += f"  • {subsection_title} [ID: {subsection_id}] ({subsection_duration} min)\n"
+                        
+                        # Add subsection questions with prominent IDs
+                        subsection_questions = subsection.get('questions', [])
+                        if subsection_questions:
+                            context += f"    Questions ({len(subsection_questions)}):\n"
+                            for k, sub_question in enumerate(subsection_questions):
+                                sub_question_id = sub_question.get('id', f'sub-question-{k}')
+                                sub_question_content = sub_question.get('content', 'No content')
+                                sub_question_time = sub_question.get('time_limit', 'No limit')
+                                context += f"      - [ID: {sub_question_id}] {sub_question_content} (Time: {sub_question_time} min)\n"
+                                
+                                # Add probes if available
+                                if sub_question.get('probes'):
+                                    probes = sub_question['probes']
+                                    if isinstance(probes, list) and probes:
+                                        # Handle both string and dict probes
+                                        probe_strings = []
+                                        for probe in probes:
+                                            if isinstance(probe, str):
+                                                probe_strings.append(probe)
+                                            elif isinstance(probe, dict):
+                                                # Extract content from probe dict
+                                                probe_strings.append(probe.get('content', str(probe)))
+                                            else:
+                                                probe_strings.append(str(probe))
+                                        context += f"        Probes: {'; '.join(probe_strings)}\n"
+                
+                context += "\n"
+            
+            return context
+        else:
+            return "No discussion guide available"
+    
+    @staticmethod
+    def _get_current_section(focus_group: Dict[str, Any]) -> str:
+        """Get current section information."""
+        moderator_position = focus_group.get('moderator_position', {})
+        discussion_guide = focus_group.get('discussionGuide', {})
+        
+        if isinstance(discussion_guide, dict) and 'sections' in discussion_guide:
+            section_index = moderator_position.get('section_index', 0)
+            sections = discussion_guide['sections']
+            
+            if section_index < len(sections):
+                current_section = sections[section_index]
+                return f"{current_section.get('title', 'Unknown')} ({current_section.get('type', 'unknown')})"
+        
+        return "Discussion in progress"
+    
+    @staticmethod
+    def format_context_for_llm(context: Dict[str, Any]) -> str:
+        """Format the context dictionary for LLM consumption."""
+        formatted_context = {}
+        
+        # Format participants context
+        participants_text = ""
+        for participant in context.get('participants_context', []):
+            ocean_traits = participant.get('personality', {}).get('ocean_traits', {})
+            participants_text += f"\n**{participant['name']}** (ID: {participant['id']})\n"
+            participants_text += f"- Demographics: {participant['demographics']['age']}, {participant['demographics']['gender']}, {participant['demographics']['occupation']}\n"
+            participants_text += f"- Location: {participant['demographics']['location']}\n"
+            participants_text += f"- Personality: {participant['personality']['description']}\n"
+            
+            if ocean_traits:
+                participants_text += f"- OCEAN Traits: "
+                traits = []
+                for trait, value in ocean_traits.items():
+                    if isinstance(value, (int, float)):
+                        traits.append(f"{trait.capitalize()}: {value}/100")
+                participants_text += ", ".join(traits) + "\n"
+            
+            goals = participant.get('personality', {}).get('goals', [])
+            if goals:
+                participants_text += f"- Goals: {', '.join(goals[:3])}\n"
+            
+            interests = participant.get('interests', '')
+            if interests:
+                participants_text += f"- Interests: {interests}\n"
+            
+            participants_text += "\n"
+        
+        # Format conversation history
+        history_text = ""
+        for msg in context.get('conversation_history', []):
+            sender_name = msg['sender_id'] if msg['sender_type'] == 'moderator' else f"Participant {msg['sender_id']}"
+            history_text += f"**{sender_name}**: {msg['content']}\n"
+        
+        # Format analytics
+        analytics = context.get('conversation_analytics', {})
+        analytics_text = f"""
+**Participation Statistics:**
+- Total messages: {analytics.get('total_messages', 0)}
+- Recent activity: {analytics.get('recent_activity', {}).get('messages_in_last_10', 0)} messages in last 10
+- Last speaker: {analytics.get('recent_activity', {}).get('last_speaker', 'unknown')}
+
+**Sentiment Analysis:**
+- Overall sentiment: {analytics.get('sentiment_analysis', {}).get('overall_sentiment', 'neutral')}
+- Agreement level: {analytics.get('sentiment_analysis', {}).get('agreement_level', 0.5):.1%}
+- Energy level: {analytics.get('sentiment_analysis', {}).get('energy_level', 0.5):.1%}
+
+**Emerging Topics:**
+{', '.join(analytics.get('topic_emergence', [])[:5])}
+"""
+        
+        formatted_context.update({
+            'focus_group_topic': context.get('focus_group_topic', 'Unknown'),
+            'focus_group_duration': context.get('focus_group_duration', 60),
+            'current_time': context.get('current_time', 0),
+            'discussion_guide_context': context.get('discussion_guide_context', 'No guide available'),
+            'current_section': context.get('current_section', 'Unknown'),
+            'participants_context': participants_text,
+            'conversation_history': history_text,
+            'conversation_analytics': analytics_text
+        })
+        
+        return formatted_context
+    
+    # ================== MULTIMODAL CONVERSATION CONTEXT METHODS ==================
+    
+    @staticmethod
+    def build_multimodal_context(focus_group_id: str, messages: Optional[List[Dict[str, Any]]] = None) -> Dict[str, Any]:
+        """
+        Build complete multimodal conversation context including text and images in proper sequence.
+        
+        Args:
+            focus_group_id: The focus group ID
+            messages: Optional list of messages (if not provided, will fetch from database)
+            
+        Returns:
+            Dictionary containing structured conversation context for LLM consumption
+        """
+        try:
+            print(f"🎯 Building multimodal context for focus group {focus_group_id}")
+            
+            # Get messages with visual context if not provided
+            if messages is None:
+                messages = FocusGroup.get_messages_with_visual_context(focus_group_id)
+            
+            # Get active visual context
+            active_visual_context = FocusGroup.get_active_visual_context(focus_group_id)
+            
+            print(f"  - Total messages: {len(messages)}")
+            print(f"  - Active visual assets: {len(active_visual_context)}")
+            
+            # Build visual timeline
+            visual_timeline = ConversationContextService._extract_visual_timeline(active_visual_context)
+            
+            # Build conversation context with images interspersed
+            conversation_context = ConversationContextService._prepare_llm_context(
+                messages, visual_timeline, focus_group_id
+            )
+            
+            # Build text-only context for backwards compatibility
+            text_context = ConversationContextService._format_text_context_simple(messages)
+            
+            result = {
+                "has_visual_context": len(active_visual_context) > 0,
+                "conversation_context": conversation_context,
+                "text_context": text_context,  # For backwards compatibility
+                "visual_timeline": visual_timeline,
+                "total_messages": len(messages),
+                "total_visual_assets": len(active_visual_context)
+            }
+            
+            print(f"✅ Built multimodal context: {len(conversation_context)} context items")
+            return result
+            
+        except Exception as e:
+            print(f"❌ Error building multimodal context: {e}")
+            raise Exception(f"Error building conversation context: {str(e)}")
+    
+    @staticmethod 
+    def _extract_visual_timeline(active_visual_context: List[Dict[str, Any]]) -> Dict[int, List[Dict[str, Any]]]:
+        """
+        Extract timeline of when each image was introduced in the conversation.
+        
+        Args:
+            active_visual_context: List of active visual asset records
+            
+        Returns:
+            Dictionary mapping sequence numbers to lists of assets activated at that point
+        """
+        visual_timeline = {}
+        
+        for asset in active_visual_context:
+            sequence = asset.get("activated_at_sequence", 0)
+            if sequence not in visual_timeline:
+                visual_timeline[sequence] = []
+            visual_timeline[sequence].append(asset)
+        
+        print(f"📸 Visual timeline: {len(visual_timeline)} activation points")
+        return visual_timeline
+    
+    @staticmethod
+    def _prepare_llm_context(
+        messages: List[Dict[str, Any]], 
+        visual_timeline: Dict[int, List[Dict[str, Any]]], 
+        focus_group_id: str
+    ) -> List[Dict[str, Any]]:
+        """
+        Prepare conversation context for LLM consumption with images interspersed in proper sequence.
+        
+        Args:
+            messages: List of messages in chronological order
+            visual_timeline: Timeline of visual asset activations
+            focus_group_id: Focus group ID for asset path resolution
+            
+        Returns:
+            List of context items including both text and image elements
+        """
+        conversation_context = []
+        
+        for i, message in enumerate(messages):
+            sequence = i + 1
+            
+            # Add text message to context
+            sender = message.get('senderId', 'Unknown')
+            text = message.get('text', '')
+            msg_type = message.get('type', 'response')
+            
+            # Format sender name
+            if msg_type == 'question':
+                formatted_sender = "MODERATOR"
+            elif msg_type == 'system':
+                formatted_sender = "SYSTEM"
+            else:
+                formatted_sender = sender
+            
+            conversation_context.append({
+                "type": "text",
+                "content": f"{formatted_sender}: {text}",
+                "sequence": sequence,
+                "message_id": message.get("_id"),
+                "sender_id": sender,
+                "message_type": msg_type
+            })
+            
+            # Add any visual assets that were activated at this sequence point
+            if sequence in visual_timeline:
+                for asset in visual_timeline[sequence]:
+                    asset_path = ConversationContextService._resolve_asset_path(
+                        focus_group_id, asset["filename"]
+                    )
+                    
+                    conversation_context.append({
+                        "type": "image",
+                        "path": asset_path,
+                        "filename": asset["filename"],
+                        "sequence": sequence,
+                        "activated_at_message_id": asset.get("activated_at_message_id"),
+                        "activation_timestamp": asset.get("activation_timestamp")
+                    })
+                    
+                    print(f"🖼️  Added image to context: {asset['filename']} at sequence {sequence}")
+        
+        return conversation_context
+    
+    @staticmethod
+    def _resolve_asset_path(focus_group_id: str, filename: str) -> str:
+        """
+        Resolve the full path to a creative asset file.
+        
+        Args:
+            focus_group_id: The focus group ID
+            filename: The asset filename
+            
+        Returns:
+            Full path to the asset file
+        """
+        base_dir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))  # Go up to backend/
+        asset_path = os.path.join(base_dir, 'uploads', f'focus-group-{focus_group_id}', filename)
+        return asset_path
+    
+    @staticmethod
+    def _format_text_context_simple(messages: List[Dict[str, Any]]) -> str:
+        """
+        Format messages as text-only context for backwards compatibility.
+        
+        Args:
+            messages: List of messages
+            
+        Returns:
+            Formatted text context string
+        """
+        if not messages:
+            return "No previous messages."
+        
+        # Limit to the most recent messages for context
+        recent_messages = messages[-50:]  # Last 50 messages
+        
+        formatted = []
+        for msg in recent_messages:
+            sender = msg.get('senderId', 'Unknown')
+            text = msg.get('text', '')
+            msg_type = msg.get('type', 'response')
+            
+            # Format differently based on message type
+            if msg_type == 'question':
+                formatted.append(f"MODERATOR ({sender}): {text}")
+            elif msg_type == 'system':
+                formatted.append(f"SYSTEM: {text}")
+            else:
+                formatted.append(f"{sender}: {text}")
+        
+        return "\n".join(formatted)
+    
+    @staticmethod
+    def get_current_visual_assets(focus_group_id: str) -> List[str]:
+        """
+        Get list of asset paths that are currently active in conversation context.
+        
+        Args:
+            focus_group_id: The focus group ID
+            
+        Returns:
+            List of full paths to currently active visual assets
+        """
+        try:
+            active_context = FocusGroup.get_active_visual_context(focus_group_id)
+            asset_paths = []
+            
+            for asset in active_context:
+                asset_path = ConversationContextService._resolve_asset_path(
+                    focus_group_id, asset["filename"]
+                )
+                asset_paths.append(asset_path)
+            
+            print(f"🎨 Current visual assets for {focus_group_id}: {len(asset_paths)} files")
+            return asset_paths
+            
+        except Exception as e:
+            print(f"❌ Error getting current visual assets: {e}")
+            return []
+    
+    @staticmethod
+    def has_visual_context(focus_group_id: str) -> bool:
+        """
+        Check if a focus group currently has any active visual context.
+        
+        Args:
+            focus_group_id: The focus group ID
+            
+        Returns:
+            True if there are active visual assets, False otherwise
+        """
+        try:
+            active_context = FocusGroup.get_active_visual_context(focus_group_id)
+            return len(active_context) > 0
+        except Exception as e:
+            print(f"❌ Error checking visual context: {e}")
+            return False

backend/app/services/conversation_decision_service.py

@@ -0,0 +1,364 @@
+"""
+Conversation Decision Service
+Uses LLM to make intelligent decisions about conversation flow, participant selection, and moderation.
+"""
+
+from typing import Dict, Any, Optional, List
+import json
+from app.services.llm_service import LLMService, LLMServiceError
+from app.services.conversation_context_service import ConversationContextService
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+
+
+class ConversationDecisionError(Exception):
+    """Exception raised for errors in conversation decision making."""
+    pass
+
+
+class ConversationDecisionService:
+    """Service for making LLM-based conversation decisions."""
+    
+    @staticmethod
+    def decide_next_action(focus_group_id: str, temperature: float = 0.7, mode: str = "ai") -> Dict[str, Any]:
+        """
+        Use LLM to decide the next action in the conversation.
+        
+        Args:
+            focus_group_id: The focus group ID
+            temperature: LLM temperature for decision making
+            mode: The conversation mode - "ai" for autonomous mode, "manual" for manual mode
+            
+        Returns:
+            Dictionary containing the decision and action details
+            
+        Raises:
+            ConversationDecisionError: If there's an issue with decision making
+        """
+        print(f"🎯 Decision request: {mode} mode for focus group {focus_group_id}")
+        
+        try:
+            # Get full conversation context
+            context = ConversationContextService.get_full_context(focus_group_id)
+            formatted_context = ConversationContextService.format_context_for_llm(context)
+            
+            # Load the appropriate prompt based on mode
+            try:
+                if mode == "manual":
+                    prompt_name = 'conversation-participant-selection'
+                else:
+                    prompt_name = 'conversation-decision-engine'
+                
+                prompt = load_prompt(prompt_name, formatted_context)
+            except PromptLoaderError as e:
+                print(f"❌ Error loading {mode} mode prompt: {str(e)}")
+                raise ConversationDecisionError(f"Error loading {mode} mode prompt: {str(e)}")
+            
+            # Get LLM decision
+            try:
+                response = LLMService.generate_content(
+                    prompt=prompt,
+                    temperature=temperature
+                )
+                
+                # Parse the JSON response
+                decision = LLMService.parse_json_response(response)
+                
+                # Validate decision structure
+                if not ConversationDecisionService._validate_decision(decision):
+                    print(f"❌ Invalid decision structure from LLM: {decision}")
+                    raise ConversationDecisionError("Invalid decision structure from LLM")
+                
+                # Log essential decision info
+                action = decision.get('action', 'unknown')
+                if action == 'participant_respond':
+                    participant_id = decision.get('details', {}).get('participant_id', 'unknown')
+                    print(f"✅ Decision: {action} for participant {participant_id}")
+                else:
+                    print(f"✅ Decision: {action}")
+                
+                return decision
+                
+            except LLMServiceError as e:
+                print(f"❌ LLM Service Error: {str(e)}")
+                raise ConversationDecisionError(f"Error getting LLM decision: {str(e)}")
+            except Exception as e:
+                print(f"❌ Unexpected error in LLM processing: {str(e)}")
+                raise ConversationDecisionError(f"Unexpected error in LLM processing: {str(e)}")
+            
+        except ConversationDecisionError:
+            # Re-raise ConversationDecisionError as-is
+            raise
+        except Exception as e:
+            print(f"❌ Unexpected error in conversation decision making: {str(e)}")
+            import traceback
+            print(f"❌ Full traceback: {traceback.format_exc()}")
+            raise ConversationDecisionError(f"Error in conversation decision making: {str(e)}")
+    
+    @staticmethod
+    def _validate_decision(decision: Dict[str, Any]) -> bool:
+        """Validate that the LLM decision has the correct structure."""
+        if not isinstance(decision, dict):
+            return False
+        
+        # Check required fields
+        required_fields = ['action', 'reasoning', 'details']
+        for field in required_fields:
+            if field not in decision:
+                return False
+        
+        # Validate optional discussion_guide_position_id field if present
+        if 'discussion_guide_position_id' in decision:
+            if not isinstance(decision['discussion_guide_position_id'], str) or not decision['discussion_guide_position_id'].strip():
+                return False
+        
+        # Check action type
+        valid_actions = ['moderator_speak', 'participant_respond', 'participant_interaction', 'probe_trigger', 'end_session']
+        if decision['action'] not in valid_actions:
+            return False
+        
+        # Validate details based on action type
+        details = decision['details']
+        if not isinstance(details, dict):
+            return False
+        
+        action = decision['action']
+        
+        if action == 'moderator_speak':
+            required_details = ['message_type', 'content']
+            return all(field in details for field in required_details)
+        
+        elif action == 'participant_respond':
+            required_details = ['participant_id', 'call_out', 'topic_context']
+            return all(field in details for field in required_details)
+        
+        elif action == 'participant_interaction':
+            required_details = ['participant_ids', 'interaction_type', 'moderator_prompt']
+            return all(field in details for field in required_details) and isinstance(details['participant_ids'], list)
+        
+        elif action == 'probe_trigger':
+            required_details = ['trigger_type', 'probe_question']
+            return all(field in details for field in required_details)
+        
+        elif action == 'end_session':
+            required_details = ['completion_reason', 'closing_message']
+            return all(field in details for field in required_details)
+        
+        return True
+    
+    @staticmethod
+    def select_next_participant(focus_group_id: str, current_topic: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to select the next participant to respond.
+        
+        Args:
+            focus_group_id: The focus group ID
+            current_topic: The current topic being discussed
+            temperature: LLM temperature for selection
+            
+        Returns:
+            Dictionary containing participant selection details
+        """
+        try:
+            decision = ConversationDecisionService.decide_next_action(focus_group_id, temperature)
+            
+            if decision['action'] == 'participant_respond':
+                return {
+                    'participant_id': decision['details']['participant_id'],
+                    'call_out': decision['details']['call_out'],
+                    'topic_context': decision['details']['topic_context'],
+                    'reasoning': decision['reasoning']
+                }
+            else:
+                # If LLM decided on a different action, return that instead
+                return {
+                    'alternative_action': decision['action'],
+                    'decision': decision
+                }
+                
+        except ConversationDecisionError as e:
+            raise e
+        except Exception as e:
+            raise ConversationDecisionError(f"Error selecting participant: {str(e)}")
+    
+    @staticmethod
+    def detect_probe_triggers(focus_group_id: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to detect if probe triggers are needed.
+        
+        Args:
+            focus_group_id: The focus group ID
+            temperature: LLM temperature for detection
+            
+        Returns:
+            Dictionary containing probe trigger information
+        """
+        try:
+            decision = ConversationDecisionService.decide_next_action(focus_group_id, temperature)
+            
+            if decision['action'] == 'probe_trigger':
+                return {
+                    'trigger_detected': True,
+                    'trigger_type': decision['details']['trigger_type'],
+                    'probe_question': decision['details']['probe_question'],
+                    'target_participants': decision['details'].get('target_participants', []),
+                    'reasoning': decision['reasoning']
+                }
+            else:
+                return {
+                    'trigger_detected': False,
+                    'alternative_action': decision['action'],
+                    'decision': decision
+                }
+                
+        except ConversationDecisionError as e:
+            raise e
+        except Exception as e:
+            raise ConversationDecisionError(f"Error detecting probe triggers: {str(e)}")
+    
+    @staticmethod
+    def generate_moderator_response(focus_group_id: str, context: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to generate appropriate moderator response.
+        
+        Args:
+            focus_group_id: The focus group ID
+            context: Additional context for the response
+            temperature: LLM temperature for generation
+            
+        Returns:
+            Dictionary containing moderator response details
+        """
+        try:
+            decision = ConversationDecisionService.decide_next_action(focus_group_id, temperature)
+            
+            if decision['action'] == 'moderator_speak':
+                return {
+                    'message_type': decision['details']['message_type'],
+                    'content': decision['details']['content'],
+                    'target_participants': decision['details'].get('target_participants', []),
+                    'reasoning': decision['reasoning']
+                }
+            else:
+                return {
+                    'alternative_action': decision['action'],
+                    'decision': decision
+                }
+                
+        except ConversationDecisionError as e:
+            raise e
+        except Exception as e:
+            raise ConversationDecisionError(f"Error generating moderator response: {str(e)}")
+    
+    @staticmethod
+    def detect_persona_interactions(focus_group_id: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to detect when personas should interact directly.
+        
+        Args:
+            focus_group_id: The focus group ID
+            temperature: LLM temperature for detection
+            
+        Returns:
+            Dictionary containing persona interaction details
+        """
+        try:
+            decision = ConversationDecisionService.decide_next_action(focus_group_id, temperature)
+            
+            if decision['action'] == 'participant_interaction':
+                return {
+                    'interaction_needed': True,
+                    'participant_ids': decision['details']['participant_ids'],
+                    'interaction_type': decision['details']['interaction_type'],
+                    'moderator_prompt': decision['details']['moderator_prompt'],
+                    'reasoning': decision['reasoning']
+                }
+            else:
+                return {
+                    'interaction_needed': False,
+                    'alternative_action': decision['action'],
+                    'decision': decision
+                }
+                
+        except ConversationDecisionError as e:
+            raise e
+        except Exception as e:
+            raise ConversationDecisionError(f"Error detecting persona interactions: {str(e)}")
+    
+    @staticmethod
+    def should_end_session(focus_group_id: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to determine if the session should end.
+        
+        Args:
+            focus_group_id: The focus group ID
+            temperature: LLM temperature for decision
+            
+        Returns:
+            Dictionary containing session ending decision
+        """
+        try:
+            decision = ConversationDecisionService.decide_next_action(focus_group_id, temperature)
+            
+            if decision['action'] == 'end_session':
+                return {
+                    'should_end': True,
+                    'completion_reason': decision['details']['completion_reason'],
+                    'closing_message': decision['details']['closing_message'],
+                    'reasoning': decision['reasoning']
+                }
+            else:
+                return {
+                    'should_end': False,
+                    'continue_action': decision['action'],
+                    'decision': decision
+                }
+                
+        except ConversationDecisionError as e:
+            raise e
+        except Exception as e:
+            raise ConversationDecisionError(f"Error determining session end: {str(e)}")
+    
+    @staticmethod
+    def get_conversation_insights(focus_group_id: str, temperature: float = 0.7) -> Dict[str, Any]:
+        """
+        Use LLM to generate insights about the current conversation state.
+        
+        Args:
+            focus_group_id: The focus group ID
+            temperature: LLM temperature for analysis
+            
+        Returns:
+            Dictionary containing conversation insights
+        """
+        try:
+            # Get conversation context
+            context = ConversationContextService.get_full_context(focus_group_id)
+            
+            # Create a specialized prompt for insights
+            insight_prompt = f"""
+            Analyze the current focus group conversation and provide insights:
+            
+            {ConversationContextService.format_context_for_llm(context)}
+            
+            Please provide insights in the following JSON format:
+            {{
+                "participation_balance": "balanced" | "unbalanced" | "needs_attention",
+                "conversation_energy": "high" | "medium" | "low",
+                "topic_engagement": "high" | "medium" | "low",
+                "sentiment_trend": "positive" | "neutral" | "negative",
+                "key_themes": ["theme1", "theme2", "theme3"],
+                "recommendations": ["rec1", "rec2", "rec3"],
+                "next_suggested_action": "specific recommendation for next step"
+            }}
+            """
+            
+            response = LLMService.generate_content(
+                prompt=insight_prompt,
+                temperature=temperature
+            )
+            
+            insights = LLMService.parse_json_response(response)
+            return insights
+            
+        except Exception as e:
+            raise ConversationDecisionError(f"Error generating conversation insights: {str(e)}")

backend/app/services/conversation_state_manager.py

@@ -0,0 +1,677 @@
+"""
+Conversation State Manager
+Manages conversation state, analytics, and tracking for autonomous focus groups.
+"""
+
+from typing import Dict, List, Any, Optional
+from datetime import datetime, timedelta
+from collections import defaultdict
+import json
+
+from app.models.focus_group import FocusGroup
+
+
+class ConversationStateManager:
+    """Manager for conversation state and analytics."""
+    
+    def __init__(self, focus_group_id: str):
+        self.focus_group_id = focus_group_id
+        self.state_cache = {}
+        self.analytics_cache = {}
+        self.cache_ttl = 60  # seconds
+        self.last_cache_update = None
+    
+    def get_conversation_state(self) -> Dict[str, Any]:
+        """
+        Get the current conversation state.
+        
+        Returns:
+            Dictionary containing conversation state information
+        """
+        try:
+            # Check cache first
+            if self._is_cache_valid():
+                return self.state_cache
+            
+            focus_group = FocusGroup.find_by_id(self.focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            # Get messages
+            messages = FocusGroup.get_messages(self.focus_group_id)
+            
+            # Calculate conversation state
+            state = {
+                "focus_group_id": self.focus_group_id,
+                "status": focus_group.get('status', 'unknown'),
+                "total_messages": len(messages),
+                "participants": focus_group.get('participants', []),
+                "created_at": focus_group.get('created_at'),
+                "updated_at": focus_group.get('updated_at'),
+                "conversation_flow": self._analyze_conversation_flow(messages),
+                "current_topic": self._get_current_topic(messages),
+                "participation_stats": self._calculate_participation_stats(messages, focus_group.get('participants', [])),
+                "conversation_health": self._assess_conversation_health(messages),
+                "moderator_position": focus_group.get('moderator_position', {}),
+                "autonomous_state": {
+                    "is_autonomous": focus_group.get('status', '').startswith('autonomous'),
+                    "started_at": focus_group.get('autonomous_started_at'),
+                    "paused_at": focus_group.get('autonomous_paused_at'),
+                    "resumed_at": focus_group.get('autonomous_resumed_at'),
+                    "ended_at": focus_group.get('autonomous_ended_at')
+                }
+            }
+            
+            # Update cache
+            self.state_cache = state
+            self.last_cache_update = datetime.utcnow()
+            
+            return state
+            
+        except Exception as e:
+            return {"error": f"Error getting conversation state: {str(e)}"}
+    
+    def get_conversation_analytics(self) -> Dict[str, Any]:
+        """
+        Get detailed conversation analytics.
+        
+        Returns:
+            Dictionary containing conversation analytics
+        """
+        try:
+            # Check cache first
+            if self._is_analytics_cache_valid():
+                return self.analytics_cache
+            
+            focus_group = FocusGroup.find_by_id(self.focus_group_id)
+            if not focus_group:
+                return {"error": "Focus group not found"}
+            
+            messages = FocusGroup.get_messages(self.focus_group_id)
+            participants = focus_group.get('participants', [])
+            
+            analytics = {
+                "focus_group_id": self.focus_group_id,
+                "overview": self._generate_overview_analytics(messages, participants),
+                "participation": self._generate_participation_analytics(messages, participants),
+                "conversation_flow": self._generate_flow_analytics(messages),
+                "sentiment_analysis": self._generate_sentiment_analytics(messages),
+                "topic_analysis": self._generate_topic_analytics(messages),
+                "timing_analysis": self._generate_timing_analytics(messages),
+                "quality_metrics": self._generate_quality_metrics(messages),
+                "recommendations": self._generate_recommendations(messages, participants)
+            }
+            
+            # Update cache
+            self.analytics_cache = analytics
+            self.last_cache_update = datetime.utcnow()
+            
+            return analytics
+            
+        except Exception as e:
+            return {"error": f"Error getting conversation analytics: {str(e)}"}
+    
+    def update_conversation_state(self, updates: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Update conversation state.
+        
+        Args:
+            updates: Dictionary of updates to apply
+            
+        Returns:
+            Dictionary containing update result
+        """
+        try:
+            # Update focus group
+            success = FocusGroup.update(self.focus_group_id, updates)
+            
+            if success:
+                # Clear cache to force refresh
+                self._clear_cache()
+                
+                return {
+                    "message": "Conversation state updated",
+                    "focus_group_id": self.focus_group_id,
+                    "updates": updates
+                }
+            else:
+                return {"error": "Failed to update conversation state"}
+                
+        except Exception as e:
+            return {"error": f"Error updating conversation state: {str(e)}"}
+    
+    def start_autonomous_mode(self) -> Dict[str, Any]:
+        """Start autonomous conversation mode."""
+        return self.update_conversation_state({
+            'status': 'ai_mode',
+            'autonomous_started_at': datetime.utcnow()
+        })
+    
+    
+    
+    def end_autonomous_mode(self, reason: str = "completed") -> Dict[str, Any]:
+        """End autonomous conversation mode."""
+        if reason == "completed":
+            status = 'completed'
+        else:
+            status = 'active'
+        return self.update_conversation_state({
+            'status': status,
+            'autonomous_ended_at': datetime.utcnow(),
+            'completion_reason': reason
+        })
+    
+    def _is_cache_valid(self) -> bool:
+        """Check if the state cache is still valid."""
+        if not self.last_cache_update or not self.state_cache:
+            return False
+        
+        elapsed = (datetime.utcnow() - self.last_cache_update).total_seconds()
+        return elapsed < self.cache_ttl
+    
+    def _is_analytics_cache_valid(self) -> bool:
+        """Check if the analytics cache is still valid."""
+        if not self.last_cache_update or not self.analytics_cache:
+            return False
+        
+        elapsed = (datetime.utcnow() - self.last_cache_update).total_seconds()
+        return elapsed < self.cache_ttl
+    
+    def _clear_cache(self):
+        """Clear all caches."""
+        self.state_cache = {}
+        self.analytics_cache = {}
+        self.last_cache_update = None
+    
+    def _analyze_conversation_flow(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Analyze the flow of conversation."""
+        if not messages:
+            return {"status": "no_messages", "flow_type": "none"}
+        
+        # Count message types
+        message_types = defaultdict(int)
+        for msg in messages:
+            message_types[msg.get('type', 'unknown')] += 1
+        
+        # Analyze flow patterns
+        recent_senders = []
+        for msg in messages[-10:]:  # Last 10 messages
+            sender = msg.get('senderId', 'unknown')
+            recent_senders.append(sender)
+        
+        # Determine flow characteristics
+        unique_recent_senders = len(set(recent_senders))
+        total_recent = len(recent_senders)
+        
+        if unique_recent_senders == 1:
+            flow_type = "monologue"
+        elif unique_recent_senders == 2:
+            flow_type = "dialogue"
+        else:
+            flow_type = "group_discussion"
+        
+        return {
+            "status": "active" if len(messages) > 0 else "inactive",
+            "flow_type": flow_type,
+            "message_types": dict(message_types),
+            "recent_participation": {
+                "unique_speakers": unique_recent_senders,
+                "total_messages": total_recent,
+                "diversity_ratio": unique_recent_senders / total_recent if total_recent > 0 else 0
+            }
+        }
+    
+    def _get_current_topic(self, messages: List[Dict[str, Any]]) -> str:
+        """Get the current topic of discussion."""
+        if not messages:
+            return "No discussion yet"
+        
+        # Get the most recent moderator question
+        for msg in reversed(messages):
+            if msg.get('senderId') == 'moderator' and msg.get('type') == 'question':
+                return msg.get('text', 'Unknown topic')[:100] + "..."
+        
+        return "General discussion"
+    
+    def _calculate_participation_stats(self, messages: List[Dict[str, Any]], participants: List[str]) -> Dict[str, Any]:
+        """Calculate participation statistics."""
+        stats = {
+            "total_participants": len(participants),
+            "active_participants": 0,
+            "participation_balance": "unknown",
+            "participant_details": {}
+        }
+        
+        if not messages:
+            return stats
+        
+        # Count messages per participant
+        participant_counts = defaultdict(int)
+        for msg in messages:
+            sender = msg.get('senderId')
+            if sender != 'moderator':
+                participant_counts[sender] += 1
+        
+        # Calculate statistics
+        active_participants = len(participant_counts)
+        stats["active_participants"] = active_participants
+        
+        # Calculate participation balance
+        if active_participants > 0:
+            message_counts = list(participant_counts.values())
+            max_messages = max(message_counts)
+            min_messages = min(message_counts)
+            
+            if max_messages - min_messages <= 2:
+                stats["participation_balance"] = "balanced"
+            elif max_messages > min_messages * 2:
+                stats["participation_balance"] = "unbalanced"
+            else:
+                stats["participation_balance"] = "moderately_balanced"
+        
+        # Detailed participant stats
+        for participant_id in participants:
+            stats["participant_details"][participant_id] = {
+                "message_count": participant_counts[participant_id],
+                "participation_percentage": (participant_counts[participant_id] / len([m for m in messages if m.get('senderId') != 'moderator'])) * 100 if any(m.get('senderId') != 'moderator' for m in messages) else 0,
+                "is_active": participant_counts[participant_id] > 0
+            }
+        
+        return stats
+    
+    def _assess_conversation_health(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Assess the health of the conversation."""
+        if not messages:
+            return {"status": "inactive", "score": 0, "indicators": []}
+        
+        health_indicators = []
+        health_score = 0
+        
+        # Check message frequency
+        recent_messages = [m for m in messages if m.get('created_at')]
+        if len(recent_messages) >= 5:
+            health_score += 25
+            health_indicators.append("active_discussion")
+        
+        # Check participation diversity
+        unique_senders = len(set(m.get('senderId') for m in messages[-10:] if m.get('senderId') != 'moderator'))
+        if unique_senders >= 3:
+            health_score += 25
+            health_indicators.append("diverse_participation")
+        
+        # Check message quality (length as proxy)
+        avg_length = sum(len(m.get('text', '')) for m in messages) / len(messages)
+        if avg_length >= 50:
+            health_score += 25
+            health_indicators.append("substantive_responses")
+        
+        # Check for questions and engagement
+        question_count = sum(1 for m in messages if '?' in m.get('text', ''))
+        if question_count >= 3:
+            health_score += 25
+            health_indicators.append("engaged_inquiry")
+        
+        # Determine overall health status
+        if health_score >= 75:
+            status = "excellent"
+        elif health_score >= 50:
+            status = "good"
+        elif health_score >= 25:
+            status = "fair"
+        else:
+            status = "poor"
+        
+        return {
+            "status": status,
+            "score": health_score,
+            "indicators": health_indicators
+        }
+    
+    def _generate_overview_analytics(self, messages: List[Dict[str, Any]], participants: List[str]) -> Dict[str, Any]:
+        """Generate overview analytics."""
+        return {
+            "total_messages": len(messages),
+            "participant_messages": len([m for m in messages if m.get('senderId') != 'moderator']),
+            "moderator_messages": len([m for m in messages if m.get('senderId') == 'moderator']),
+            "total_participants": len(participants),
+            "active_participants": len(set(m.get('senderId') for m in messages if m.get('senderId') != 'moderator')),
+            "avg_message_length": sum(len(m.get('text', '')) for m in messages) / len(messages) if messages else 0,
+            "conversation_duration": self._calculate_conversation_duration(messages)
+        }
+    
+    def _generate_participation_analytics(self, messages: List[Dict[str, Any]], participants: List[str]) -> Dict[str, Any]:
+        """Generate participation analytics."""
+        participant_stats = defaultdict(lambda: {
+            "message_count": 0,
+            "total_words": 0,
+            "avg_message_length": 0,
+            "participation_percentage": 0
+        })
+        
+        total_participant_messages = 0
+        
+        for msg in messages:
+            sender = msg.get('senderId')
+            if sender != 'moderator':
+                text = msg.get('text', '')
+                word_count = len(text.split())
+                
+                participant_stats[sender]["message_count"] += 1
+                participant_stats[sender]["total_words"] += word_count
+                total_participant_messages += 1
+        
+        # Calculate percentages and averages
+        for sender, stats in participant_stats.items():
+            if stats["message_count"] > 0:
+                stats["avg_message_length"] = stats["total_words"] / stats["message_count"]
+                stats["participation_percentage"] = (stats["message_count"] / total_participant_messages) * 100
+        
+        return {
+            "participant_stats": dict(participant_stats),
+            "participation_balance": self._calculate_participation_balance(participant_stats),
+            "dominant_participants": self._identify_dominant_participants(participant_stats),
+            "quiet_participants": self._identify_quiet_participants(participants, participant_stats)
+        }
+    
+    def _generate_flow_analytics(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Generate conversation flow analytics."""
+        if not messages:
+            return {"interaction_patterns": [], "flow_quality": "none"}
+        
+        # Analyze interaction patterns
+        interaction_patterns = []
+        for i in range(len(messages) - 1):
+            current_sender = messages[i].get('senderId')
+            next_sender = messages[i + 1].get('senderId')
+            
+            if current_sender != next_sender:
+                interaction_patterns.append({
+                    "from": current_sender,
+                    "to": next_sender,
+                    "type": "response" if next_sender != 'moderator' else "moderation"
+                })
+        
+        return {
+            "interaction_patterns": interaction_patterns[-10:],  # Last 10 interactions
+            "flow_quality": self._assess_flow_quality(interaction_patterns),
+            "turn_taking_analysis": self._analyze_turn_taking(messages)
+        }
+    
+    def _generate_sentiment_analytics(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Generate sentiment analytics."""
+        # Simple sentiment analysis based on keywords
+        positive_words = ['good', 'great', 'excellent', 'love', 'like', 'amazing', 'wonderful', 'agree', 'yes']
+        negative_words = ['bad', 'terrible', 'hate', 'dislike', 'awful', 'horrible', 'disagree', 'no', 'problem']
+        
+        sentiment_scores = []
+        for msg in messages:
+            text = msg.get('text', '').lower()
+            positive_count = sum(1 for word in positive_words if word in text)
+            negative_count = sum(1 for word in negative_words if word in text)
+            
+            if positive_count > negative_count:
+                sentiment_scores.append(1)
+            elif negative_count > positive_count:
+                sentiment_scores.append(-1)
+            else:
+                sentiment_scores.append(0)
+        
+        avg_sentiment = sum(sentiment_scores) / len(sentiment_scores) if sentiment_scores else 0
+        
+        return {
+            "overall_sentiment": "positive" if avg_sentiment > 0.2 else "negative" if avg_sentiment < -0.2 else "neutral",
+            "sentiment_trend": self._calculate_sentiment_trend(sentiment_scores),
+            "sentiment_distribution": {
+                "positive": sentiment_scores.count(1),
+                "neutral": sentiment_scores.count(0),
+                "negative": sentiment_scores.count(-1)
+            }
+        }
+    
+    def _generate_topic_analytics(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Generate topic analytics."""
+        # Simple topic extraction based on word frequency
+        from collections import Counter
+        
+        all_words = []
+        for msg in messages:
+            text = msg.get('text', '').lower()
+            words = [word.strip('.,!?;:"()[]{}') for word in text.split()]
+            all_words.extend(words)
+        
+        # Filter out common words
+        stop_words = {'the', 'a', 'an', 'and', 'or', 'but', 'in', 'on', 'at', 'to', 'for', 'of', 'with', 'by', 'is', 'are', 'was', 'were', 'i', 'you', 'it', 'we', 'they', 'this', 'that'}
+        filtered_words = [word for word in all_words if len(word) > 3 and word not in stop_words]
+        
+        word_freq = Counter(filtered_words)
+        
+        return {
+            "top_topics": [{"word": word, "frequency": freq} for word, freq in word_freq.most_common(10)],
+            "topic_diversity": len(word_freq),
+            "vocabulary_richness": len(set(all_words)) / len(all_words) if all_words else 0
+        }
+    
+    def _generate_timing_analytics(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Generate timing analytics."""
+        if not messages:
+            return {"avg_response_time": 0, "message_frequency": 0}
+        
+        # Calculate response times (simplified)
+        response_times = []
+        for i in range(1, len(messages)):
+            current_time = messages[i].get('created_at')
+            previous_time = messages[i - 1].get('created_at')
+            
+            if current_time and previous_time:
+                if isinstance(current_time, datetime) and isinstance(previous_time, datetime):
+                    response_time = (current_time - previous_time).total_seconds()
+                    response_times.append(response_time)
+        
+        avg_response_time = sum(response_times) / len(response_times) if response_times else 0
+        
+        return {
+            "avg_response_time": avg_response_time,
+            "message_frequency": len(messages) / max(1, self._calculate_conversation_duration(messages)),
+            "response_time_distribution": self._categorize_response_times(response_times)
+        }
+    
+    def _generate_quality_metrics(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Generate quality metrics."""
+        if not messages:
+            return {"engagement_score": 0, "depth_score": 0, "quality_score": 0}
+        
+        # Calculate engagement (questions, exclamations, etc.)
+        engagement_indicators = 0
+        for msg in messages:
+            text = msg.get('text', '')
+            if '?' in text:
+                engagement_indicators += 1
+            if '!' in text:
+                engagement_indicators += 1
+        
+        engagement_score = min(100, (engagement_indicators / len(messages)) * 100)
+        
+        # Calculate depth (message length, detail)
+        total_words = sum(len(msg.get('text', '').split()) for msg in messages)
+        avg_words_per_message = total_words / len(messages)
+        depth_score = min(100, avg_words_per_message * 2)  # Rough scoring
+        
+        # Overall quality score
+        quality_score = (engagement_score + depth_score) / 2
+        
+        return {
+            "engagement_score": engagement_score,
+            "depth_score": depth_score,
+            "quality_score": quality_score,
+            "total_words": total_words,
+            "avg_words_per_message": avg_words_per_message
+        }
+    
+    def _generate_recommendations(self, messages: List[Dict[str, Any]], participants: List[str]) -> List[str]:
+        """Generate recommendations based on analytics."""
+        recommendations = []
+        
+        # Analyze participation
+        participant_counts = defaultdict(int)
+        for msg in messages:
+            if msg.get('senderId') != 'moderator':
+                participant_counts[msg.get('senderId')] += 1
+        
+        # Check for quiet participants
+        quiet_participants = [p for p in participants if participant_counts[p] == 0]
+        if quiet_participants:
+            recommendations.append(f"Encourage participation from {len(quiet_participants)} quiet participants")
+        
+        # Check for dominant participants
+        if participant_counts:
+            max_messages = max(participant_counts.values())
+            total_messages = sum(participant_counts.values())
+            if max_messages > total_messages * 0.4:
+                recommendations.append("One participant is dominating the conversation - consider moderating")
+        
+        # Check conversation depth
+        if messages:
+            avg_length = sum(len(msg.get('text', '')) for msg in messages) / len(messages)
+            if avg_length < 30:
+                recommendations.append("Responses are quite brief - consider asking follow-up questions")
+        
+        # Check for questions
+        question_count = sum(1 for msg in messages if '?' in msg.get('text', ''))
+        if question_count < len(messages) * 0.1:
+            recommendations.append("Low level of inquiry - encourage more questions between participants")
+        
+        return recommendations[:5]  # Return top 5 recommendations
+    
+    def _calculate_conversation_duration(self, messages: List[Dict[str, Any]]) -> float:
+        """Calculate conversation duration in minutes."""
+        if not messages:
+            return 0
+        
+        timestamps = [msg.get('created_at') for msg in messages if msg.get('created_at')]
+        if not timestamps:
+            return 0
+        
+        # Filter out non-datetime objects
+        valid_timestamps = [ts for ts in timestamps if isinstance(ts, datetime)]
+        if len(valid_timestamps) < 2:
+            return 0
+        
+        duration = (max(valid_timestamps) - min(valid_timestamps)).total_seconds() / 60
+        return duration
+    
+    def _calculate_participation_balance(self, participant_stats: Dict[str, Dict[str, Any]]) -> str:
+        """Calculate participation balance."""
+        if not participant_stats:
+            return "no_data"
+        
+        message_counts = [stats["message_count"] for stats in participant_stats.values()]
+        if not message_counts:
+            return "no_messages"
+        
+        max_messages = max(message_counts)
+        min_messages = min(message_counts)
+        
+        if max_messages - min_messages <= 2:
+            return "balanced"
+        elif max_messages > min_messages * 3:
+            return "highly_unbalanced"
+        else:
+            return "moderately_unbalanced"
+    
+    def _identify_dominant_participants(self, participant_stats: Dict[str, Dict[str, Any]]) -> List[str]:
+        """Identify dominant participants."""
+        if not participant_stats:
+            return []
+        
+        # Find participants with >40% of messages
+        dominant = []
+        for participant, stats in participant_stats.items():
+            if stats["participation_percentage"] > 40:
+                dominant.append(participant)
+        
+        return dominant
+    
+    def _identify_quiet_participants(self, participants: List[str], participant_stats: Dict[str, Dict[str, Any]]) -> List[str]:
+        """Identify quiet participants."""
+        quiet = []
+        for participant in participants:
+            if participant not in participant_stats or participant_stats[participant]["message_count"] == 0:
+                quiet.append(participant)
+        
+        return quiet
+    
+    def _assess_flow_quality(self, interaction_patterns: List[Dict[str, Any]]) -> str:
+        """Assess the quality of conversation flow."""
+        if not interaction_patterns:
+            return "none"
+        
+        # Check for varied interactions
+        unique_interactions = len(set((p["from"], p["to"]) for p in interaction_patterns))
+        
+        if unique_interactions >= len(interaction_patterns) * 0.7:
+            return "excellent"
+        elif unique_interactions >= len(interaction_patterns) * 0.5:
+            return "good"
+        elif unique_interactions >= len(interaction_patterns) * 0.3:
+            return "fair"
+        else:
+            return "poor"
+    
+    def _analyze_turn_taking(self, messages: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Analyze turn-taking patterns."""
+        if len(messages) < 3:
+            return {"pattern": "insufficient_data"}
+        
+        # Count consecutive messages from same sender
+        consecutive_counts = []
+        current_sender = None
+        current_count = 0
+        
+        for msg in messages:
+            sender = msg.get('senderId')
+            if sender == current_sender:
+                current_count += 1
+            else:
+                if current_count > 0:
+                    consecutive_counts.append(current_count)
+                current_sender = sender
+                current_count = 1
+        
+        if current_count > 0:
+            consecutive_counts.append(current_count)
+        
+        avg_consecutive = sum(consecutive_counts) / len(consecutive_counts) if consecutive_counts else 0
+        
+        return {
+            "pattern": "healthy" if avg_consecutive <= 2 else "some_domination" if avg_consecutive <= 4 else "poor_turn_taking",
+            "avg_consecutive_messages": avg_consecutive,
+            "max_consecutive": max(consecutive_counts) if consecutive_counts else 0
+        }
+    
+    def _calculate_sentiment_trend(self, sentiment_scores: List[int]) -> str:
+        """Calculate sentiment trend."""
+        if len(sentiment_scores) < 3:
+            return "insufficient_data"
+        
+        # Compare first half vs second half
+        mid_point = len(sentiment_scores) // 2
+        first_half_avg = sum(sentiment_scores[:mid_point]) / mid_point if mid_point > 0 else 0
+        second_half_avg = sum(sentiment_scores[mid_point:]) / (len(sentiment_scores) - mid_point)
+        
+        if second_half_avg > first_half_avg + 0.2:
+            return "improving"
+        elif second_half_avg < first_half_avg - 0.2:
+            return "declining"
+        else:
+            return "stable"
+    
+    def _categorize_response_times(self, response_times: List[float]) -> Dict[str, int]:
+        """Categorize response times."""
+        if not response_times:
+            return {"fast": 0, "medium": 0, "slow": 0}
+        
+        fast = sum(1 for t in response_times if t < 30)  # < 30 seconds
+        medium = sum(1 for t in response_times if 30 <= t < 120)  # 30s - 2min
+        slow = sum(1 for t in response_times if t >= 120)  # > 2min
+        
+        return {"fast": fast, "medium": medium, "slow": slow}

backend/app/services/customer_data_service.py

@@ -0,0 +1,168 @@
+"""
+Customer Data Service for parsing uploaded files using LlamaParse.
+Handles file upload, parsing to markdown, and cleanup operations.
+"""
+
+import os
+import uuid
+import shutil
+import tempfile
+from typing import List, Optional
+from werkzeug.datastructures import FileStorage
+
+try:
+    from llama_cloud_services import LlamaParse
+except ImportError:
+    LlamaParse = None
+
+
+class CustomerDataServiceError(Exception):
+    """Exception raised for errors in customer data processing."""
+    pass
+
+
+class CustomerDataService:
+    """Service for handling customer data upload and parsing."""
+    
+    def __init__(self, api_key: str = "llx-HhMSCmLjYAuK7FcxJ0yBxAP4t4JY0tKx7XLyZGHJJWiUFZuX"):
+        """Initialize the service with LlamaParse API key."""
+        if not LlamaParse:
+            raise CustomerDataServiceError("llama-cloud-services package not installed")
+        
+        self.api_key = api_key
+        self.base_dir = os.path.join(os.path.dirname(__file__), "..", "..", "persona_data")
+        
+        # Ensure base directory exists
+        os.makedirs(self.base_dir, exist_ok=True)
+        
+        # Initialize LlamaParse in premium mode
+        self.parser = LlamaParse(
+            api_key=self.api_key,
+            result_type="markdown",
+            premium_mode=True,  # Enable premium mode for best accuracy
+            parsing_instruction="Extract all customer data including demographics, behaviors, preferences, purchase history, and any other relevant customer information. Preserve data structure and relationships where possible.",
+            num_workers=4,
+            verbose=True,
+            language="en"
+        )
+    
+    def generate_session_id(self) -> str:
+        """Generate a unique session ID for this upload session."""
+        return str(uuid.uuid4())
+    
+    def upload_and_parse_files(self, files: List[FileStorage]) -> str:
+        """
+        Upload files and parse them using LlamaParse.
+        
+        Args:
+            files: List of uploaded files
+            
+        Returns:
+            session_id: Unique identifier for this upload session
+            
+        Raises:
+            CustomerDataServiceError: If upload or parsing fails
+        """
+        if not files or len(files) == 0:
+            raise CustomerDataServiceError("No files provided")
+        
+        session_id = self.generate_session_id()
+        session_dir = os.path.join(self.base_dir, session_id)
+        
+        try:
+            # Create session directory
+            os.makedirs(session_dir, exist_ok=True)
+            
+            # Save uploaded files
+            uploaded_files = []
+            for file in files:
+                if file.filename and file.filename.strip():
+                    # Secure filename
+                    filename = f"{session_id}_{file.filename}"
+                    file_path = os.path.join(session_dir, filename)
+                    file.save(file_path)
+                    uploaded_files.append(file_path)
+            
+            if not uploaded_files:
+                raise CustomerDataServiceError("No valid files uploaded")
+            
+            # Parse files using LlamaParse
+            parsed_documents = self.parser.load_data(uploaded_files)
+            
+            # Save parsed markdown files
+            for i, document in enumerate(parsed_documents):
+                markdown_filename = f"{session_id}_parsed_{i+1}.md"
+                markdown_path = os.path.join(session_dir, markdown_filename)
+                with open(markdown_path, 'w', encoding='utf-8') as f:
+                    f.write(document.text)
+            
+            return session_id
+            
+        except Exception as e:
+            # Clean up on error
+            if os.path.exists(session_dir):
+                shutil.rmtree(session_dir, ignore_errors=True)
+            raise CustomerDataServiceError(f"Failed to process files: {str(e)}")
+    
+    def get_parsed_markdown_content(self, session_id: str) -> Optional[str]:
+        """
+        Get all parsed markdown content for a session.
+        
+        Args:
+            session_id: The session identifier
+            
+        Returns:
+            Combined markdown content from all parsed files, or None if not found
+        """
+        if not session_id:
+            return None
+            
+        session_dir = os.path.join(self.base_dir, session_id)
+        if not os.path.exists(session_dir):
+            return None
+        
+        combined_content = []
+        
+        # Find all markdown files for this session
+        for filename in os.listdir(session_dir):
+            if filename.endswith('.md') and 'parsed' in filename:
+                file_path = os.path.join(session_dir, filename)
+                try:
+                    with open(file_path, 'r', encoding='utf-8') as f:
+                        content = f.read().strip()
+                        if content:
+                            combined_content.append(f"## {filename}\n\n{content}")
+                except Exception as e:
+                    continue  # Skip files that can't be read
+        
+        if combined_content:
+            return "\n\n---\n\n".join(combined_content)
+        
+        return None
+    
+    def cleanup_session(self, session_id: str) -> bool:
+        """
+        Clean up all files for a session.
+        
+        Args:
+            session_id: The session identifier
+            
+        Returns:
+            True if cleanup was successful, False otherwise
+        """
+        if not session_id:
+            return False
+            
+        session_dir = os.path.join(self.base_dir, session_id)
+        if os.path.exists(session_dir):
+            try:
+                shutil.rmtree(session_dir)
+                return True
+            except Exception:
+                return False
+        
+        return True  # Nothing to clean up
+
+
+# Global service instance
+customer_data_service = CustomerDataService()

backend/app/services/focus_group_response_service.py

@@ -0,0 +1,560 @@
+"""
+Focus Group Response Service
+This service handles generating realistic responses from personas during focus group discussions.
+"""
+
+from typing import Dict, Any, Optional, List, Union
+import json
+import random
+import os
+from .llm_service import LLMService, LLMServiceError
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+
+class FocusGroupResponseError(Exception):
+    """Exception raised for errors in the focus group response generation process."""
+    pass
+
+
+def generate_persona_response(
+    persona: Dict[str, Any],
+    current_topic: str,
+    previous_messages: List[Dict[str, Any]],
+    temperature: float = 0.7,
+    focus_group_id: Optional[str] = None
+) -> str:
+    """
+    Generate a response from a persona in a focus group discussion.
+    Now integrates visual context when available.
+    
+    Args:
+        persona: The persona data (personality, traits, etc.)
+        current_topic: The current question or topic being discussed
+        previous_messages: List of previous messages in the discussion
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        focus_group_id: Optional focus group ID for visual context integration
+        
+    Returns:
+        A string containing the persona's response
+        
+    Raises:
+        FocusGroupResponseError: If there's an issue with the response generation
+    """
+    try:
+        print(f"🎭 Generating persona response for {persona.get('name', 'Unknown')}")
+        print(f"  - focus_group_id: {focus_group_id}")
+        print(f"  - current_topic: {current_topic[:50]}...")
+        
+        # Import LLMService at the top to avoid scoping issues
+        from app.services.llm_service import LLMService
+        
+        # Check for visual context if focus_group_id is provided
+        has_visual_context = False
+        multimodal_context = None
+        
+        if focus_group_id:
+            try:
+                from app.services.conversation_context_service import ConversationContextService
+                has_visual_context = ConversationContextService.has_visual_context(focus_group_id)
+                
+                if has_visual_context:
+                    print(f"🎨 Visual context detected, building multimodal context...")
+                    multimodal_context = ConversationContextService.build_multimodal_context(
+                        focus_group_id, previous_messages
+                    )
+                    print(f"🎨 Built context with {multimodal_context['total_visual_assets']} visual assets")
+                else:
+                    print(f"📝 No visual context, using standard generation")
+            except Exception as e:
+                print(f"⚠️  Error checking visual context, falling back to standard generation: {e}")
+                has_visual_context = False
+        
+        # Determine the appropriate response length based on persona and context
+        length_preference = _determine_response_length_preference(
+            persona, previous_messages, current_topic
+        )
+        
+        # Get length-specific instructions
+        length_instructions = _get_length_specific_instructions(length_preference)
+        
+        # Extract relevant persona details for the prompt
+        persona_details = _format_persona_details(persona)
+        
+        # If we have visual context, use contextual generation
+        if has_visual_context and multimodal_context:
+            print(f"🎨 Using contextual generation with visual context")
+            
+            # Load and format the contextual response prompt
+            try:
+                prompt = load_prompt('focus-group-response', {
+                    'persona_details': persona_details,
+                    'current_topic': current_topic,
+                    'previous_messages': multimodal_context['text_context'],  # Use text fallback
+                    'length_instructions': length_instructions,
+                    'is_creative_review': True,  # Flag to indicate visual context available
+                    'creative_instructions': """
+
+VISUAL CONTEXT AVAILABLE:
+You are participating in a focus group discussion where visual materials have been shown. The images in your conversation context are part of the ongoing discussion. Please provide your authentic reaction and feedback based on your personality, background, and preferences, taking into account both the conversation history and any visual materials you can see.
+
+Consider:
+- Your first impression of any visuals shown
+- How the visual materials relate to the discussion topic  
+- Any specific elements that catch your attention
+- How the visuals might appeal to people like you
+- Any suggestions or concerns you might have
+- The ongoing conversation context
+
+Be genuine and specific in your feedback, drawing on your personal experiences and preferences.
+"""
+                })
+            except PromptLoaderError as e:
+                raise FocusGroupResponseError(f"Error loading contextual response prompt: {str(e)}")
+            
+            # Generate response using contextual conversation method
+            response = LLMService.generate_contextual_response(
+                prompt=prompt,
+                conversation_context=multimodal_context['conversation_context'],
+                temperature=temperature
+            )
+            
+            print(f"✅ Generated contextual response with visual context")
+            
+        else:
+            print(f"📝 Using standard generation (no visual context)")
+            
+            # Format the previous messages for context (standard approach)
+            formatted_messages = _format_previous_messages(previous_messages)
+            
+            # Load and format the standard response prompt
+            try:
+                prompt = load_prompt('focus-group-response', {
+                    'persona_details': persona_details,
+                    'current_topic': current_topic,
+                    'previous_messages': formatted_messages,
+                    'length_instructions': length_instructions
+                })
+            except PromptLoaderError as e:
+                raise FocusGroupResponseError(f"Error loading response prompt: {str(e)}")
+            
+            # Generate the standard response
+            response = LLMService.generate_content(
+                prompt=prompt,
+                temperature=temperature
+            )
+            
+            print(f"✅ Generated standard response")
+        
+        return response.strip()
+        
+    except LLMServiceError as e:
+        raise FocusGroupResponseError(f"Error generating persona response: {str(e)}")
+    except Exception as e:
+        raise FocusGroupResponseError(f"Unexpected error in persona response generation: {str(e)}")
+
+def _format_persona_details(persona: Dict[str, Any]) -> str:
+    """Format persona details for the prompt."""
+    details = []
+    
+    # Basic demographics
+    details.append(f"Name: {persona.get('name', 'Unknown')}")
+    details.append(f"Age: {persona.get('age', 'Unknown')}")
+    details.append(f"Gender: {persona.get('gender', 'Unknown')}")
+    details.append(f"Occupation: {persona.get('occupation', 'Unknown')}")
+    details.append(f"Education: {persona.get('education', 'Unknown')}")
+    details.append(f"Location: {persona.get('location', 'Unknown')}")
+    
+    # Personality characteristics
+    details.append(f"Personality: {persona.get('personality', 'Not specified')}")
+    
+    # OCEAN traits if available
+    ocean = persona.get('oceanTraits', {})
+    if ocean:
+        traits = []
+        if 'openness' in ocean:
+            traits.append(f"Openness: {ocean['openness']}/100")
+        if 'conscientiousness' in ocean:
+            traits.append(f"Conscientiousness: {ocean['conscientiousness']}/100")
+        if 'extraversion' in ocean:
+            traits.append(f"Extraversion: {ocean['extraversion']}/100")
+        if 'agreeableness' in ocean:
+            traits.append(f"Agreeableness: {ocean['agreeableness']}/100")
+        if 'neuroticism' in ocean:
+            traits.append(f"Neuroticism: {ocean['neuroticism']}/100")
+        
+        if traits:
+            details.append("OCEAN Traits:")
+            details.extend([f"- {trait}" for trait in traits])
+    
+    # Goals, frustrations, motivations
+    if 'goals' in persona and persona['goals']:
+        details.append("Goals:")
+        details.extend([f"- {goal}" for goal in persona['goals']])
+        
+    if 'frustrations' in persona and persona['frustrations']:
+        details.append("Frustrations:")
+        details.extend([f"- {frustration}" for frustration in persona['frustrations']])
+        
+    if 'motivations' in persona and persona['motivations']:
+        details.append("Motivations:")
+        details.extend([f"- {motivation}" for motivation in persona['motivations']])
+    
+    # Think, feel, do
+    tfd = persona.get('thinkFeelDo', {})
+    if tfd:
+        if 'thinks' in tfd and tfd['thinks']:
+            details.append("Thinks:")
+            details.extend([f"- {thought}" for thought in tfd['thinks']])
+            
+        if 'feels' in tfd and tfd['feels']:
+            details.append("Feels:")
+            details.extend([f"- {feeling}" for feeling in tfd['feels']])
+            
+        if 'does' in tfd and tfd['does']:
+            details.append("Does:")
+            details.extend([f"- {action}" for action in tfd['does']])
+    
+    # Join all details with line breaks
+    return "\n".join(details)
+
+def _format_previous_messages(messages: List[Dict[str, Any]]) -> str:
+    """Format previous messages for context."""
+    if not messages:
+        return "No previous messages."
+    
+    # Limit to the most recent messages for context
+    recent_messages = messages[-50:]  # Last 50 messages
+    
+    formatted = []
+    for msg in recent_messages:
+        sender = msg.get('senderId', 'Unknown')
+        text = msg.get('text', '')
+        msg_type = msg.get('type', 'response')
+        
+        # Format differently based on message type
+        if msg_type == 'question':
+            formatted.append(f"MODERATOR ({sender}): {text}")
+        elif msg_type == 'system':
+            formatted.append(f"SYSTEM: {text}")
+        else:
+            formatted.append(f"{sender}: {text}")
+    
+    return "\n".join(formatted)
+
+
+def _determine_response_length_preference(
+    persona: Dict[str, Any], 
+    previous_messages: List[Dict[str, Any]], 
+    current_topic: str
+) -> str:
+    """
+    Determine the preferred response length based on persona traits and context.
+    
+    Args:
+        persona: The persona data
+        previous_messages: List of previous messages in the discussion
+        current_topic: The current question or topic being discussed
+        
+    Returns:
+        Response length preference: 'short', 'medium', or 'long'
+    """
+    # Base probabilities for response lengths
+    short_prob = 0.25   # 25% chance of short responses
+    medium_prob = 0.50  # 50% chance of medium responses  
+    long_prob = 0.25    # 25% chance of long responses
+    
+    # Adjust based on persona extraversion (if available)
+    ocean_traits = persona.get('oceanTraits', {})
+    if 'extraversion' in ocean_traits:
+        extraversion = ocean_traits['extraversion'] / 100.0  # Normalize to 0-1
+        
+        # High extraversion = more likely to give longer responses
+        # Low extraversion = more likely to give shorter responses
+        if extraversion > 0.7:  # High extraversion
+            short_prob *= 0.6    # Reduce short response probability
+            medium_prob *= 0.9   # Slightly reduce medium
+            long_prob *= 1.8     # Increase long response probability
+        elif extraversion < 0.3:  # Low extraversion  
+            short_prob *= 1.6    # Increase short response probability
+            medium_prob *= 1.1   # Slightly increase medium
+            long_prob *= 0.5     # Reduce long response probability
+    
+    # Adjust based on communication preferences
+    comm_prefs = persona.get('communicationPreferences', '').lower()
+    if 'brief' in comm_prefs or 'concise' in comm_prefs or 'direct' in comm_prefs:
+        short_prob *= 1.4
+        medium_prob *= 1.1
+        long_prob *= 0.6
+    elif 'detailed' in comm_prefs or 'verbose' in comm_prefs or 'elaborate' in comm_prefs:
+        short_prob *= 0.7
+        medium_prob *= 0.9
+        long_prob *= 1.5
+    
+    # Analyze recent message context
+    if previous_messages:
+        recent_messages = previous_messages[-5:]  # Last 5 messages
+        recent_lengths = []
+        
+        for msg in recent_messages:
+            text = msg.get('text', '')
+            word_count = len(text.split())
+            recent_lengths.append(word_count)
+        
+        if recent_lengths:
+            avg_recent_length = sum(recent_lengths) / len(recent_lengths)
+            
+            # If recent messages are short, sometimes match the brevity
+            if avg_recent_length < 10:  # Very short recent messages
+                short_prob *= 1.3
+                medium_prob *= 1.0
+                long_prob *= 0.7
+            # If recent messages are long, sometimes provide contrast with shorter response
+            elif avg_recent_length > 50:  # Long recent messages
+                short_prob *= 1.2
+                medium_prob *= 1.1  
+                long_prob *= 0.8
+    
+    # Consider topic complexity (simple heuristic)
+    topic_words = current_topic.split()
+    if len(topic_words) > 15 or current_topic.count('?') > 1:
+        # Complex topics may warrant longer responses
+        short_prob *= 0.8
+        medium_prob *= 1.0
+        long_prob *= 1.3
+    
+    # Normalize probabilities
+    total_prob = short_prob + medium_prob + long_prob
+    short_prob /= total_prob
+    medium_prob /= total_prob
+    long_prob /= total_prob
+    
+    # Select length based on weighted random choice
+    rand = random.random()
+    if rand < short_prob:
+        return 'short'
+    elif rand < short_prob + medium_prob:
+        return 'medium'
+    else:
+        return 'long'
+
+
+def _get_length_specific_instructions(length_preference: str) -> str:
+    """
+    Get length-specific instructions for the LLM prompt.
+    
+    Args:
+        length_preference: The preferred response length ('short', 'medium', 'long')
+        
+    Returns:
+        Instructions specific to the response length
+    """
+    if length_preference == 'short':
+        return """
+RESPONSE LENGTH: Provide a SHORT response (1-8 words or a brief phrase).
+Examples of appropriate short responses:
+- "Absolutely!"
+- "I disagree."
+- "That's interesting."
+- "Not really."
+- "Exactly my point."
+- "Makes sense to me."
+- "I'm not sure about that."
+
+Keep it natural and conversational, but brief. Sometimes a simple reaction or acknowledgment is all that's needed.
+"""
+    elif length_preference == 'medium':
+        return """
+RESPONSE LENGTH: Provide a MEDIUM response (1-3 sentences).
+This should be conversational but not overly detailed. Share your perspective clearly and concisely.
+Example length: "I think that's a great point about mobile payments. I've had similar experiences with apps that make checkout too complicated."
+"""
+    else:  # long
+        return """
+RESPONSE LENGTH: Provide a LONGER response (2-4 sentences or 1-2 short paragraphs).
+Feel free to elaborate on your thoughts, share personal examples, or explore different aspects of the topic.
+This is your chance to provide more detailed insights and personal anecdotes.
+"""
+
+
+def generate_creative_review_response(
+    persona: Dict[str, Any],
+    current_topic: str,
+    creative_asset_path: str,
+    previous_messages: List[Dict[str, Any]],
+    focus_group_id: str,
+    temperature: float = 0.7
+) -> str:
+    """
+    Generate a response from a persona for a creative review activity with image context.
+    
+    Args:
+        persona: The persona data (personality, traits, etc.)
+        current_topic: The current question or topic being discussed
+        creative_asset_path: Path to the creative asset image file
+        previous_messages: List of previous messages in the discussion
+        focus_group_id: The focus group ID for asset path resolution
+        temperature: Controls randomness in generation (0.0 = deterministic, 1.0 = creative)
+        
+    Returns:
+        A string containing the persona's response to the creative asset
+        
+    Raises:
+        FocusGroupResponseError: If there's an issue with the response generation
+    """
+    try:
+        print(f"🎨 CREATIVE REVIEW RESPONSE DEBUG:")
+        print(f"  - persona: {persona.get('name', 'Unknown')}")
+        print(f"  - current_topic: {current_topic}")
+        print(f"  - creative_asset_path: {creative_asset_path}")
+        print(f"  - focus_group_id: {focus_group_id}")
+        print(f"  - temperature: {temperature}")
+        
+        # Determine the appropriate response length
+        length_preference = _determine_response_length_preference(
+            persona, previous_messages, current_topic
+        )
+        print(f"  - length_preference: {length_preference}")
+        
+        # Get length-specific instructions
+        length_instructions = _get_length_specific_instructions(length_preference)
+        
+        # Extract relevant persona details for the prompt
+        persona_details = _format_persona_details(persona)
+        
+        # Format the previous messages for context
+        formatted_messages = _format_previous_messages(previous_messages)
+        
+        # Construct the full path to the creative asset
+        if not os.path.isabs(creative_asset_path):
+            # Files are stored in focus group subdirectories: uploads/focus-group-{id}/filename
+            base_dir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))  # Go up to backend/
+            full_asset_path = os.path.join(base_dir, 'uploads', f'focus-group-{focus_group_id}', creative_asset_path)
+        else:
+            full_asset_path = creative_asset_path
+        
+        print(f"  - full_asset_path: {full_asset_path}")
+        print(f"  - asset_exists: {os.path.exists(full_asset_path)}")
+        
+        # Verify the asset file exists
+        if not os.path.exists(full_asset_path):
+            print(f"❌ Creative asset not found at: {full_asset_path}")
+            # List available files in uploads directory for debugging
+            uploads_dir = os.path.dirname(full_asset_path)
+            if os.path.exists(uploads_dir):
+                available_files = os.listdir(uploads_dir)
+                print(f"  - Available files in uploads: {available_files}")
+            raise FocusGroupResponseError(f"Creative asset not found: {full_asset_path}")
+        
+        # Load and format the creative response prompt
+        try:
+            prompt = load_prompt('focus-group-response', {
+                'persona_details': persona_details,
+                'current_topic': current_topic,
+                'previous_messages': formatted_messages,
+                'length_instructions': length_instructions,
+                'is_creative_review': True,
+                'creative_instructions': """
+                
+CREATIVE ASSET CONTEXT:
+You are now viewing a creative asset (image) that is being shown to you as part of this focus group discussion. 
+Please provide your authentic reaction and feedback based on your personality, background, and preferences.
+
+Consider:
+- Your first impression of the visual
+- How it relates to the discussion topic
+- Any specific elements that catch your attention
+- How it might appeal to people like you
+- Any suggestions or concerns you might have
+
+Be genuine and specific in your feedback, drawing on your personal experiences and preferences.
+"""
+            })
+        except PromptLoaderError as e:
+            raise FocusGroupResponseError(f"Error loading creative response prompt: {str(e)}")
+        
+        # Generate the response using multimodal capabilities
+        print(f"🎨 Calling LLMService.generate_multimodal_content...")
+        print(f"  - prompt length: {len(prompt)} characters")
+        print(f"  - image_paths: {[full_asset_path]}")
+        print(f"  - temperature: {temperature}")
+        
+        response = LLMService.generate_multimodal_content(
+            prompt=prompt,
+            image_paths=[full_asset_path],
+            temperature=temperature
+        )
+        
+        print(f"✅ Creative review response generated successfully")
+        print(f"  - response length: {len(response)} characters")
+        print(f"  - response preview: {response[:100]}...")
+        
+        return response.strip()
+        
+    except LLMServiceError as e:
+        raise FocusGroupResponseError(f"Error generating creative review response: {str(e)}")
+    except Exception as e:
+        raise FocusGroupResponseError(f"Unexpected error in creative review response generation: {str(e)}")
+
+
+def get_upload_folder_path(focus_group_id: str) -> str:
+    """
+    Get the upload folder path for a focus group.
+    
+    Args:
+        focus_group_id: The focus group ID
+        
+    Returns:
+        The full path to the upload folder
+    """
+    base_dir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))  # Go up to backend/
+    return os.path.join(base_dir, 'uploads', f'focus-group-{focus_group_id}')
+
+
+def is_creative_review_activity(activity_or_question: Dict[str, Any]) -> bool:
+    """
+    Check if an activity/question is a creative review type.
+    
+    Args:
+        activity_or_question: The activity or question object
+        
+    Returns:
+        True if it's a creative review activity, False otherwise
+    """
+    return activity_or_question.get('type') == 'creative_review'
+
+
+def extract_asset_filename_from_content(content: str) -> Optional[str]:
+    """
+    Extract asset filename from creative review activity content.
+    
+    Args:
+        content: The activity content string
+        
+    Returns:
+        The asset filename if found, None otherwise
+    """
+    # Look for patterns like "asset: filename.jpg" or similar
+    import re
+    
+    # Try to find asset filename patterns in the content
+    patterns = [
+        # Match quoted filenames (most specific pattern first)
+        r"titled\s+['\"]([^'\"]+\.(jpg|jpeg|png))['\"]",  # "titled 'filename.jpg'"
+        r"asset\s+['\"]([^'\"]+\.(jpg|jpeg|png))['\"]",  # "asset 'filename.jpg'"
+        r"image\s+['\"]([^'\"]+\.(jpg|jpeg|png))['\"]",  # "image 'filename.jpg'"
+        r"['\"]([a-zA-Z0-9_\-]+\.(jpg|jpeg|png))['\"]",  # Any quoted filename
+        # Match focus group asset pattern without quotes
+        r'(fg-[a-f0-9]+-[a-f0-9]{32}\.(jpg|jpeg|png))',  # fg-{id}-{uuid}.{ext}
+        # Other patterns
+        r'asset:\s*([^\s]+\.(jpg|jpeg|png))',
+        r'image:\s*([^\s]+\.(jpg|jpeg|png))',
+        r'file:\s*([^\s]+\.(jpg|jpeg|png))',
+        r'([a-zA-Z0-9_-]+\.(jpg|jpeg|png))'
+    ]
+    
+    for pattern in patterns:
+        match = re.search(pattern, content, re.IGNORECASE)
+        if match:
+            # Return the first capture group (the filename)
+            return match.group(1)
+    
+    return None

backend/app/services/focus_group_service.py

@@ -0,0 +1,275 @@
+"""
+Focus Group Service for Synthetic Society
+This service provides functionality for generating discussion guides
+and other focus group related operations using the LLM service.
+"""
+
+from app.services.llm_service import LLMService
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+from app.utils.discussion_guide_schema import DiscussionGuideValidator
+from app.models.focus_group import FocusGroup
+from typing import Dict, Any, Optional, List, Union
+import json
+import time
+import logging
+import os
+
+# Set up logging
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+class FocusGroupService:
+    """Service for focus group operations."""
+    
+    @staticmethod
+    def generate_discussion_guide(
+        focus_group_name: str,
+        research_brief: str,
+        discussion_topics: str,
+        duration: int = 60,
+        temperature: float = 0.7,
+        max_retries: int = 3,
+        focus_group_id: Optional[str] = None
+    ) -> Dict[str, Any]:
+        """
+        Generate a focus group discussion guide using the LLM with retry logic.
+        
+        Args:
+            focus_group_name: The name of the focus group
+            research_brief: The research objectives and context
+            discussion_topics: Key topics to be covered in the discussion
+            duration: Duration of the focus group in minutes
+            temperature: Controls randomness in generation
+            max_retries: Maximum number of retry attempts
+            focus_group_id: Optional focus group ID to check for uploaded assets
+            
+        Returns:
+            A structured JSON discussion guide (dict)
+            
+        Raises:
+            Exception: If all retry attempts fail
+        """
+        logger.info(f"Starting discussion guide generation for '{focus_group_name}' (duration: {duration}min, topics: {discussion_topics})")
+        
+        # Calculate approximate section times based on duration
+        total_minutes = int(duration)
+        intro_time = max(5, int(total_minutes * 0.1))
+        warmup_time = max(5, int(total_minutes * 0.15))
+        main_topics_time = max(20, int(total_minutes * 0.5))
+        conclusion_time = max(5, int(total_minutes * 0.1))
+        remaining_time = total_minutes - (intro_time + warmup_time + main_topics_time + conclusion_time)
+        
+        # Adjust main topics time to account for remaining time
+        main_topics_time += remaining_time
+        
+        # Calculate content scaling parameters based on duration
+        if total_minutes <= 45:
+            duration_category = "short"
+            recommended_main_topics = min(2, len([topic.strip() for topic in discussion_topics.split(',')]))
+            questions_per_warmup = 2
+            questions_per_subsection = 2
+            include_creative_exercises = False
+            probe_questions_per_main = 1
+        elif total_minutes <= 75:
+            duration_category = "medium"
+            recommended_main_topics = min(3, len([topic.strip() for topic in discussion_topics.split(',')]))
+            questions_per_warmup = 3
+            questions_per_subsection = 3
+            include_creative_exercises = True
+            probe_questions_per_main = 2
+        else:  # 76+ minutes
+            duration_category = "long"
+            recommended_main_topics = min(4, len([topic.strip() for topic in discussion_topics.split(',')]))
+            questions_per_warmup = 4
+            questions_per_subsection = 4
+            include_creative_exercises = True
+            probe_questions_per_main = 3
+        
+        # Parse topics into a list
+        topic_list = [topic.strip() for topic in discussion_topics.split(',')]
+        
+        # Check for uploaded creative assets if focus_group_id is provided
+        uploaded_assets = []
+        if focus_group_id:
+            try:
+                # DEBUG: Check if focus group exists and log its data
+                focus_group_doc = FocusGroup.find_by_id(focus_group_id)
+                if focus_group_doc:
+                    logger.info(f"Found focus group document: {focus_group_id}")
+                    logger.info(f"Focus group keys: {list(focus_group_doc.keys()) if focus_group_doc else 'None'}")
+                    if 'uploaded_assets' in focus_group_doc:
+                        logger.info(f"Raw uploaded_assets in document: {focus_group_doc['uploaded_assets']}")
+                    else:
+                        logger.warning(f"No 'uploaded_assets' key found in focus group document")
+                else:
+                    logger.error(f"Focus group document not found: {focus_group_id}")
+                
+                uploaded_assets = FocusGroup.get_uploaded_assets(focus_group_id)
+                logger.info(f"Found {len(uploaded_assets)} uploaded assets for focus group {focus_group_id}")
+                if uploaded_assets:
+                    logger.info(f"Asset details: {uploaded_assets}")
+            except Exception as e:
+                logger.error(f"Could not retrieve assets for focus group {focus_group_id}: {e}")
+                import traceback
+                logger.error(f"Asset retrieval traceback: {traceback.format_exc()}")
+        
+        # Load and format the discussion guide prompt
+        try:
+            # Prepare template variables
+            template_vars = {
+                'focus_group_name': focus_group_name,
+                'research_brief': research_brief,
+                'discussion_topics': ', '.join(topic_list),
+                'duration': duration,
+                'intro_time': intro_time,
+                'warmup_time': warmup_time,
+                'main_topics_time': main_topics_time,
+                'conclusion_time': conclusion_time,
+                'duration_category': duration_category,
+                'recommended_main_topics': recommended_main_topics,
+                'questions_per_warmup': questions_per_warmup,
+                'questions_per_subsection': questions_per_subsection,
+                'include_creative_exercises': include_creative_exercises,
+                'probe_questions_per_main': probe_questions_per_main,
+                'uploaded_assets': uploaded_assets,
+                'has_assets': len(uploaded_assets) > 0,
+                'asset_count': len(uploaded_assets),
+                # Create a formatted list of asset filenames for the LLM
+                'uploaded_asset_list': '\n'.join([f"- {asset.get('filename', 'unknown')} ({asset.get('original_filename', 'unknown')})" for asset in uploaded_assets]) if uploaded_assets else 'No assets uploaded',
+                # Jinja2-style template variables to avoid conflicts with Python formatting
+                'jinja_if_has_assets': '{% if has_assets %}' if len(uploaded_assets) > 0 else '',
+                'jinja_else': '{% else %}' if len(uploaded_assets) == 0 else '',
+                'jinja_endif': '{% endif %}'
+            }
+            
+            # DEBUG: Log template variables before prompt generation
+            logger.info("=== DEBUG: TEMPLATE VARIABLES ===")
+            logger.info(f"has_assets: {template_vars['has_assets']}")
+            logger.info(f"asset_count: {template_vars['asset_count']}")
+            logger.info(f"uploaded_asset_list: {template_vars['uploaded_asset_list']}")
+            logger.info(f"jinja_if_has_assets: {template_vars['jinja_if_has_assets']}")
+            logger.info(f"jinja_else: {template_vars['jinja_else']}")
+            logger.info(f"jinja_endif: {template_vars['jinja_endif']}")
+            
+            prompt = load_prompt('discussion-guide-generation', template_vars)
+            logger.info(f"Successfully loaded discussion guide prompt template")
+            
+            # DEBUG: Log the complete prompt to verify asset information is included
+            logger.info("=== DEBUG: COMPLETE PROMPT BEING SENT TO LLM ===")
+            logger.info(f"Prompt length: {len(prompt)} characters")
+            logger.info(f"Assets in template variables: {len(uploaded_assets)} assets")
+            if uploaded_assets:
+                logger.info(f"Asset details: {[{'filename': a.get('filename'), 'original': a.get('original_filename')} for a in uploaded_assets]}")
+            
+            # Log sections around creative assets to verify template population
+            if "CREATIVE ASSETS REQUIREMENTS" in prompt:
+                creative_section_start = prompt.find("CREATIVE ASSETS REQUIREMENTS")
+                creative_section_end = prompt.find("BEST PRACTICES:", creative_section_start)
+                if creative_section_end == -1:
+                    creative_section_end = creative_section_start + 1000
+                creative_section = prompt[creative_section_start:creative_section_end]
+                logger.info("=== CREATIVE ASSETS SECTION IN PROMPT ===")
+                logger.info(creative_section)
+            else:
+                logger.warning("CREATIVE ASSETS REQUIREMENTS section not found in prompt!")
+            
+            logger.info("=== END DEBUG PROMPT ===")
+        except PromptLoaderError as e:
+            error_msg = f"Error loading discussion guide prompt: {str(e)}"
+            logger.error(error_msg)
+            raise Exception(error_msg)
+        
+        # Retry logic with exponential backoff
+        last_error = None
+        for attempt in range(1, max_retries + 1):
+            try:
+                logger.info(f"Discussion guide generation attempt {attempt}/{max_retries}")
+                
+                # Generate content using LLM
+                response = LLMService.generate_content(
+                    prompt=prompt,
+                    temperature=temperature,
+                    max_tokens=16000  # Use a much higher token limit to avoid truncation
+                )
+                
+                logger.info(f"Received LLM response (length: {len(response)} chars)")
+                
+                # Clean up the response to remove code fences if present
+                clean_response = response.strip()
+                if clean_response.startswith("```json"):
+                    clean_response = clean_response[7:].strip()
+                elif clean_response.startswith("```"):
+                    clean_response = clean_response[3:].strip()
+                    
+                # Remove trailing code fence if present
+                if clean_response.endswith("```"):
+                    clean_response = clean_response[:-3].strip()
+                
+                logger.info(f"Cleaned response (length: {len(clean_response)} chars)")
+                
+                # Try to parse as JSON
+                try:
+                    guide_json = json.loads(clean_response)
+                    logger.info(f"Successfully parsed JSON response")
+                    
+                    # Validate the JSON structure
+                    is_valid, validation_errors = DiscussionGuideValidator.validate_json_structure(guide_json)
+                    
+                    if is_valid:
+                        # Validate creative review activities if assets were uploaded
+                        if uploaded_assets and len(uploaded_assets) > 0:
+                            creative_review_count = 0
+                            sections = guide_json.get('sections', [])
+                            
+                            # Count creative_review activities across all sections
+                            for section in sections:
+                                activities = section.get('activities', [])
+                                for activity in activities:
+                                    if activity.get('type') == 'creative_review':
+                                        creative_review_count += 1
+                                
+                                # Also check in subsections
+                                subsections = section.get('subsections', [])
+                                for subsection in subsections:
+                                    activities = subsection.get('activities', [])
+                                    for activity in activities:
+                                        if activity.get('type') == 'creative_review':
+                                            creative_review_count += 1
+                            
+                            logger.info(f"Found {creative_review_count} creative_review activities for {len(uploaded_assets)} uploaded assets")
+                            
+                            # If no creative review activities were generated, log a warning but continue
+                            if creative_review_count == 0:
+                                logger.warning(f"WARNING: No creative_review activities generated despite {len(uploaded_assets)} uploaded assets!")
+                            elif creative_review_count < len(uploaded_assets):
+                                logger.warning(f"WARNING: Only {creative_review_count} creative_review activities generated for {len(uploaded_assets)} assets")
+                        
+                        logger.info(f"Discussion guide generation successful on attempt {attempt}/{max_retries}")
+                        logger.info(f"Generated guide has {len(guide_json.get('sections', []))} sections")
+                        return guide_json
+                    else:
+                        error_msg = f"Generated JSON failed validation: {validation_errors}"
+                        logger.warning(error_msg)
+                        last_error = Exception(error_msg)
+                        
+                except json.JSONDecodeError as e:
+                    error_msg = f"Failed to parse generated response as JSON: {str(e)}"
+                    logger.warning(error_msg)
+                    logger.debug(f"Raw response that failed to parse: {clean_response[:500]}...")
+                    last_error = Exception(error_msg)
+                    
+            except Exception as e:
+                error_msg = f"LLM service error during generation: {str(e)}"
+                logger.warning(error_msg)
+                last_error = e
+            
+            # If this wasn't the last attempt, wait before retrying (exponential backoff)
+            if attempt < max_retries:
+                wait_time = 2 ** (attempt - 1)  # 1, 2, 4 seconds
+                logger.info(f"Attempt {attempt} failed, waiting {wait_time} seconds before retry...")
+                time.sleep(wait_time)
+        
+        # All attempts failed
+        final_error_msg = f"Discussion guide generation failed after {max_retries} attempts. Last error: {str(last_error)}"
+        logger.error(final_error_msg)
+        raise Exception(final_error_msg)

backend/app/services/image_description_service.py

@@ -0,0 +1,164 @@
+"""
+Image Description Service
+This service generates detailed AI-powered descriptions of creative assets for focus group research.
+It helps distinguish between multiple images in the conversation context.
+"""
+
+import os
+import logging
+from typing import Optional
+from PIL import Image
+from app.services.llm_service import LLMService, LLMServiceError
+from app.services.conversation_context_service import ConversationContextService
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+
+logger = logging.getLogger(__name__)
+
+class ImageDescriptionError(Exception):
+    """Exception raised for errors in image description generation."""
+    pass
+
+class ImageDescriptionService:
+    """Service for generating AI-powered descriptions of creative assets."""
+    
+    @staticmethod
+    def generate_description(focus_group_id: str, asset_filename: str) -> str:
+        """
+        Generate a detailed AI description of a creative asset image.
+        
+        Args:
+            focus_group_id: The focus group ID containing the asset
+            asset_filename: The filename of the asset to describe
+            
+        Returns:
+            A detailed description of the image
+            
+        Raises:
+            ImageDescriptionError: If description generation fails
+        """
+        try:
+            print(f"🎨 DESCRIPTION: Generating AI description for {asset_filename}")
+            
+            # Resolve the full path to the asset
+            asset_path = ConversationContextService._resolve_asset_path(focus_group_id, asset_filename)
+            print(f"🔍 DESCRIPTION: Resolved asset path: {asset_path}")
+            
+            # Check if file exists
+            if not os.path.exists(asset_path):
+                print(f"❌ DESCRIPTION: File does not exist at path: {asset_path}")
+                # List files in the directory to help debug
+                asset_dir = os.path.dirname(asset_path)
+                if os.path.exists(asset_dir):
+                    files_in_dir = os.listdir(asset_dir)
+                    print(f"🔍 DESCRIPTION: Files in directory {asset_dir}: {files_in_dir}")
+                else:
+                    print(f"❌ DESCRIPTION: Directory does not exist: {asset_dir}")
+                raise ImageDescriptionError(f"Asset file not found: {asset_path}")
+            
+            # Verify the image can be loaded (optional validation)
+            try:
+                image = Image.open(asset_path)
+                print(f"🖼️  DESCRIPTION: Validated image {asset_filename} ({image.size[0]}x{image.size[1]})")
+                image.close()  # Close the image since we're passing the path to LLM
+            except Exception as e:
+                raise ImageDescriptionError(f"Failed to validate image {asset_filename}: {str(e)}")
+            
+            # Load the description prompt
+            try:
+                prompt = load_prompt('image-description', {})
+                print(f"📝 DESCRIPTION: Loaded description prompt ({len(prompt)} chars)")
+            except PromptLoaderError as e:
+                raise ImageDescriptionError(f"Failed to load description prompt: {str(e)}")
+            
+            # Generate description using multimodal LLM
+            try:
+                print(f"🚀 DESCRIPTION: Calling LLM service with image: {asset_path}")
+                description = LLMService.generate_multimodal_content(
+                    prompt=prompt,
+                    image_paths=[asset_path],
+                    temperature=0.7
+                )
+                print(f"✅ DESCRIPTION: LLM returned description ({len(description)} chars): {description[:100]}...")
+                return description.strip()
+                
+            except LLMServiceError as e:
+                raise ImageDescriptionError(f"LLM description generation failed: {str(e)}")
+            
+        except Exception as e:
+            print(f"⚠️  DESCRIPTION: LLM service failed, generating fallback description: {str(e)}")
+            
+            # Generate fallback description based on image info
+            try:
+                # We already validated the image exists earlier, so we can safely open it
+                image = Image.open(asset_path)
+                width, height = image.size
+                image.close()
+                
+                # Create a simple but useful fallback description
+                fallback_description = f"a {width}x{height} pixel marketing advertisement image"
+                
+                print(f"✅ DESCRIPTION: Generated fallback description: '{fallback_description}'")
+                return fallback_description
+                
+            except Exception as fallback_error:
+                error_msg = f"Failed to generate description for {asset_filename}: LLM failed ({str(e)}) and fallback failed ({str(fallback_error)})"
+                print(f"❌ DESCRIPTION: {error_msg}")
+                raise ImageDescriptionError(error_msg)
+    
+    @staticmethod
+    def enhance_creative_review_question(original_question: str, asset_filename: str, description: str) -> str:
+        """
+        Enhance a creative review question by incorporating AI-generated image description.
+        
+        Args:
+            original_question: The original question text
+            asset_filename: The asset filename being referenced
+            description: The AI-generated description of the image
+            
+        Returns:
+            Enhanced question text with detailed visual description
+        """
+        try:
+            print(f"🔧 ENHANCEMENT: Enhancing question for {asset_filename}")
+            print(f"🔧 Original: {original_question[:100]}...")
+            print(f"🔧 Description: {description[:100]}...")
+            
+            # Look for the asset filename reference in the question
+            if asset_filename in original_question:
+                # Find the filename reference and enhance it
+                filename_pattern = f"'{asset_filename}'"
+                enhanced_reference = f"'{asset_filename}' - {description}"
+                
+                enhanced_question = original_question.replace(filename_pattern, enhanced_reference)
+                
+                print(f"✅ ENHANCEMENT: Enhanced question: {enhanced_question[:150]}...")
+                return enhanced_question
+            else:
+                # If filename not found in expected format, try other patterns
+                import re
+                
+                # Try to find quoted filename patterns
+                patterns = [
+                    (f'"{asset_filename}"', f'"{asset_filename}" - {description}'),
+                    (f"titled '{asset_filename}'", f"titled '{asset_filename}' - {description}"),
+                    (f'titled "{asset_filename}"', f'titled "{asset_filename}" - {description}'),
+                    (asset_filename, f"{asset_filename} - {description}")
+                ]
+                
+                enhanced_question = original_question
+                for old_pattern, new_pattern in patterns:
+                    if old_pattern in enhanced_question:
+                        enhanced_question = enhanced_question.replace(old_pattern, new_pattern)
+                        print(f"✅ ENHANCEMENT: Enhanced with pattern '{old_pattern}': {enhanced_question[:150]}...")
+                        return enhanced_question
+                
+                # If no patterns match, append description at the end
+                enhanced_question = f"{original_question} The image shows {description}."
+                print(f"⚠️  ENHANCEMENT: Appended description to end: {enhanced_question[:150]}...")
+                return enhanced_question
+                
+        except Exception as e:
+            error_msg = f"Failed to enhance question for {asset_filename}: {str(e)}"
+            print(f"❌ ENHANCEMENT: {error_msg}")
+            # Return original question if enhancement fails
+            return original_question

backend/app/services/key_theme_service.py

@@ -0,0 +1,408 @@
+"""
+Key Theme Generation Service
+This service provides functions for generating key themes from focus group discussions.
+"""
+
+import time
+import logging
+import re
+from typing import Dict, Any, List, Optional
+from app.services.llm_service import LLMService, LLMServiceError
+from app.utils.prompt_loader import load_prompt, PromptLoaderError
+from app.models.focus_group import FocusGroup
+from app.models.persona import Persona
+
+class KeyThemeServiceError(Exception):
+    """Exception raised for errors in key theme generation."""
+    pass
+
+class KeyThemeService:
+    """Service for generating key themes from focus group discussions."""
+    
+    @staticmethod
+    def generate_key_themes(
+        focus_group_id: str,
+        temperature: float = 0.7
+    ) -> List[Dict[str, str]]:
+        """
+        Generate key themes from a focus group discussion.
+        
+        Args:
+            focus_group_id: The ID of the focus group
+            temperature: Controls randomness in generation (default: 0.7)
+            
+        Returns:
+            A list of key theme objects with title and description fields
+            
+        Raises:
+            KeyThemeServiceError: If there's an issue with the generation process
+        """
+        logger = logging.getLogger(__name__)
+        logger.info(f"Starting key theme generation for focus group {focus_group_id} with temperature {temperature}")
+        
+        try:
+            # Get the focus group
+            focus_group = FocusGroup.find_by_id(focus_group_id)
+            if not focus_group:
+                raise KeyThemeServiceError(f"Focus group not found with ID: {focus_group_id}")
+            
+            # Get all messages from the focus group
+            messages = FocusGroup.get_messages(focus_group_id)
+            if not messages:
+                raise KeyThemeServiceError("No messages found in this focus group")
+            
+            logger.info(f"Found {len(messages)} messages in focus group {focus_group_id}")
+            
+            # Get all participants (personas) in the focus group
+            participants_data = []
+            if 'participants' in focus_group and focus_group['participants']:
+                for persona_id in focus_group['participants']:
+                    try:
+                        persona = Persona.find_by_id(persona_id)
+                        if persona:
+                            participants_data.append(persona)
+                    except Exception as e:
+                        print(f"Error fetching participant {persona_id}: {e}")
+            
+            # Generate key themes using LLM
+            return KeyThemeService._extract_themes_from_discussion(
+                messages=messages,
+                participants=participants_data,
+                discussion_guide=focus_group.get('discussionGuide', ''),
+                temperature=temperature
+            )
+            
+        except Exception as e:
+            raise KeyThemeServiceError(f"Error generating key themes: {str(e)}")
+    
+    @staticmethod
+    def _extract_themes_from_discussion(
+        messages: List[Dict[str, Any]],
+        participants: List[Dict[str, Any]],
+        discussion_guide: str,
+        temperature: float = 0.7
+    ) -> List[Dict[str, str]]:
+        """
+        Extract key themes from a discussion using LLM.
+        
+        Args:
+            messages: List of discussion messages
+            participants: List of participant personas
+            discussion_guide: The discussion guide for the focus group
+            temperature: Controls randomness in generation
+            
+        Returns:
+            A list of key theme objects with title and description
+            
+        Raises:
+            KeyThemeServiceError: If there's an issue with the LLM processing
+        """
+        logger = logging.getLogger(__name__)
+        logger.info(f"Beginning theme extraction from {len(messages)} messages")
+        
+        try:
+            # Load and prepare the prompt for the LLM
+            try:
+                prompt = KeyThemeService._build_theme_extraction_prompt(
+                    messages=messages,
+                    participants=participants,
+                    discussion_guide=discussion_guide
+                )
+                logger.debug("Successfully loaded and built theme extraction prompt")
+            except PromptLoaderError as e:
+                logger.error(f"Failed to load theme extraction prompt: {str(e)}")
+                raise KeyThemeServiceError(f"Error loading theme extraction prompt: {str(e)}")
+            
+            # Load system prompt
+            try:
+                system_prompt = load_prompt('key-theme-system')
+                logger.debug("Successfully loaded system prompt")
+            except PromptLoaderError as e:
+                logger.error(f"Failed to load system prompt: {str(e)}")
+                raise KeyThemeServiceError(f"Error loading system prompt: {str(e)}")
+            
+            # Call the LLM to generate themes with retry logic
+            max_retries = 3
+            last_error = None
+            logger.info(f"Starting LLM theme generation with maximum {max_retries} attempts")
+            
+            for attempt in range(max_retries):
+                attempt_num = attempt + 1
+                logger.info(f"Attempt {attempt_num}/{max_retries}: Calling LLM for theme generation")
+                
+                try:
+                    themes = LLMService.generate_structured_array(
+                        prompt=prompt,
+                        temperature=temperature,
+                        system_prompt=system_prompt
+                    )
+                    
+                    logger.info(f"Attempt {attempt_num}/{max_retries}: LLM call successful, received {len(themes)} themes")
+                    
+                    # Validate the response structure
+                    validated_themes = []
+                    for theme in themes:
+                        if isinstance(theme, dict) and 'title' in theme and 'description' in theme:
+                            validated_theme = {
+                                'title': theme['title'],
+                                'description': theme['description']
+                            }
+                            # Add quotes if present
+                            if 'quotes' in theme and isinstance(theme['quotes'], list):
+                                # Validate and clean quotes format, extracting message IDs
+                                validated_quotes = []
+                                for quote in theme['quotes']:
+                                    if isinstance(quote, str) and quote.strip():
+                                        quote_data = KeyThemeService._parse_quote_with_message_id(quote.strip())
+                                        
+                                        # Validate that the quote exists in the original messages
+                                        if KeyThemeService._validate_quote_exists(quote_data, messages):
+                                            validated_quotes.append(quote_data)
+                                        else:
+                                            logger.warning(f"Quote validation failed for theme '{theme.get('title', 'Unknown')}': {quote[:100]}...")
+                                
+                                validated_theme['quotes'] = validated_quotes
+                            else:
+                                validated_theme['quotes'] = []
+                            
+                            validated_themes.append(validated_theme)
+                    
+                    logger.info(f"Theme generation completed successfully with {len(validated_themes)} validated themes")
+                    return validated_themes
+                    
+                except LLMServiceError as e:
+                    last_error = e
+                    error_message = str(e).lower()
+                    
+                    logger.warning(f"Attempt {attempt_num}/{max_retries}: LLM call failed with error: {str(e)}")
+                    
+                    # Check if this is a retryable error (Google API internal errors)
+                    if "500" in error_message or "internal error" in error_message:
+                        if attempt < max_retries - 1:
+                            # Wait before retrying (exponential backoff)
+                            wait_time = 2 ** attempt  # 1s, 2s, 4s
+                            logger.info(f"Retryable error detected. Waiting {wait_time} seconds before retry {attempt_num + 1}/{max_retries}")
+                            time.sleep(wait_time)
+                            continue
+                        else:
+                            logger.error(f"Retryable error detected but max retries ({max_retries}) reached")
+                    else:
+                        logger.error(f"Non-retryable error detected: {str(e)}")
+                    
+                    # If it's not a retryable error or we've exhausted retries, re-raise
+                    raise KeyThemeServiceError(f"LLM error: {str(e)}")
+            
+            # If we've exhausted all retries, raise the last error
+            logger.error(f"All {max_retries} attempts failed. Final error: {str(last_error)}")
+            raise KeyThemeServiceError(f"LLM error after {max_retries} attempts: {str(last_error)}")
+                
+        except Exception as e:
+            raise KeyThemeServiceError(f"Error extracting themes: {str(e)}")
+    
+    @staticmethod
+    def _build_theme_extraction_prompt(
+        messages: List[Dict[str, Any]],
+        participants: List[Dict[str, Any]],
+        discussion_guide: str
+    ) -> str:
+        """
+        Build the prompt for theme extraction.
+        
+        Args:
+            messages: List of discussion messages
+            participants: List of participant personas
+            discussion_guide: The discussion guide for the focus group
+            
+        Returns:
+            A formatted prompt string for the LLM
+        """
+        # Format the discussion messages with IDs for quote tracking
+        formatted_messages = []
+        for msg in messages:
+            sender_id = msg.get('senderId', '')
+            sender_name = "AI Moderator" if sender_id == "moderator" else f"Participant {sender_id}"
+            
+            # Find the participant name if available
+            for participant in participants:
+                participant_id = participant.get('_id', '') or participant.get('id', '')
+                if participant_id == sender_id:
+                    sender_name = participant.get('name', sender_name)
+                    break
+            
+            text = msg.get('text', '')
+            message_id = msg.get('id', '') or msg.get('_id', '')
+            
+            # Include message ID in the formatted message for quote tracking
+            formatted_messages.append(f"[MSG_ID:{message_id}] {sender_name}: {text}")
+        
+        # Format the participant profiles
+        formatted_profiles = []
+        for participant in participants:
+            name = participant.get('name', 'Unknown')
+            age = participant.get('age', 'Unknown')
+            occupation = participant.get('occupation', 'Unknown')
+            background = participant.get('background', '')
+            
+            profile = f"Name: {name}\nAge: {age}\nOccupation: {occupation}"
+            if background:
+                profile += f"\nBackground: {background}"
+            
+            formatted_profiles.append(profile)
+        
+        # Join formatted profiles and messages with newlines
+        profiles_text = "\n".join(formatted_profiles)
+        messages_text = "\n".join(formatted_messages)
+        
+        # Load and format the theme extraction prompt
+        try:
+            prompt = load_prompt('key-theme-extraction', {
+                'discussion_guide': discussion_guide,
+                'profiles_text': profiles_text,
+                'messages_text': messages_text
+            })
+        except PromptLoaderError as e:
+            raise KeyThemeServiceError(f"Error loading theme extraction prompt: {str(e)}")
+        
+        return prompt
+
+    @staticmethod
+    def _parse_quote_with_message_id(quote: str) -> dict:
+        """
+        Parse a quote string to extract message ID, speaker, and text.
+        
+        Expected format: "[MSG_ID:message_id] [Speaker Name]: quote text"
+        
+        Args:
+            quote: The quote string to parse
+            
+        Returns:
+            A dictionary with 'message_id', 'speaker', 'text', and 'original' fields
+        """
+        logger = logging.getLogger(__name__)
+        
+        # Initialize default structure
+        quote_data = {
+            'message_id': None,
+            'speaker': None,
+            'text': quote,
+            'original': quote
+        }
+        
+        try:
+            # Try to parse format: [MSG_ID:message_id] [Speaker Name]: quote text (legacy with brackets)
+            msg_id_pattern_brackets = r'^\[MSG_ID:([^\]]+)\]\s*\[([^\]]+)\]:\s*(.*)$'
+            match = re.match(msg_id_pattern_brackets, quote)
+            
+            if match:
+                quote_data['message_id'] = match.group(1)
+                quote_data['speaker'] = match.group(2)
+                quote_data['text'] = match.group(3)
+                logger.debug(f"Successfully parsed quote with message ID (bracketed format): {quote_data['message_id']}")
+                return quote_data
+            
+            # Try to parse format: [MSG_ID:message_id] Speaker Name: quote text (current LLM format)
+            msg_id_pattern = r'^\[MSG_ID:([^\]]+)\]\s*([^:]+):\s*(.*)$'
+            match = re.match(msg_id_pattern, quote)
+            
+            if match:
+                quote_data['message_id'] = match.group(1)
+                quote_data['speaker'] = match.group(2).strip()
+                quote_data['text'] = match.group(3)
+                logger.debug(f"Successfully parsed quote with message ID (standard format): {quote_data['message_id']}")
+                return quote_data
+            
+            # Fallback: Try legacy format [Speaker Name]: quote text
+            legacy_pattern = r'^\[([^\]]+)\]:\s*(.*)$'
+            legacy_match = re.match(legacy_pattern, quote)
+            
+            if legacy_match:
+                quote_data['speaker'] = legacy_match.group(1)
+                quote_data['text'] = legacy_match.group(2)
+                logger.warning(f"Quote using legacy format without message ID: {quote[:50]}...")
+                return quote_data
+            
+            # If no pattern matches, log warning and return as-is
+            logger.warning(f"Quote does not match expected format: {quote[:50]}...")
+            return quote_data
+            
+        except Exception as e:
+            logger.error(f"Error parsing quote '{quote[:50]}...': {str(e)}")
+            return quote_data
+
+    @staticmethod
+    def _validate_quote_exists(quote_data: dict, messages: List[Dict[str, Any]]) -> bool:
+        """
+        Validate that a quote actually exists in the original messages.
+        
+        Args:
+            quote_data: The parsed quote data with message_id, speaker, text, etc.
+            messages: List of original discussion messages
+            
+        Returns:
+            True if the quote can be validated, False otherwise
+        """
+        logger = logging.getLogger(__name__)
+        
+        quote_text = quote_data.get('text', '').strip()
+        message_id = quote_data.get('message_id')
+        
+        if not quote_text:
+            logger.warning("Quote validation failed: empty quote text")
+            return False
+        
+        # Strategy 1: Direct message ID lookup (most reliable)
+        if message_id:
+            target_message = None
+            for msg in messages:
+                msg_id = msg.get('id', '') or msg.get('_id', '')
+                if msg_id == message_id:
+                    target_message = msg
+                    break
+            
+            if target_message:
+                msg_text = target_message.get('text', '')
+                # Check if quote text exists in the target message
+                if quote_text.lower() in msg_text.lower() or msg_text.lower() in quote_text.lower():
+                    logger.debug(f"Quote validated via message ID {message_id}")
+                    return True
+                else:
+                    logger.warning(f"Quote text doesn't match message ID {message_id}: quote='{quote_text[:50]}...', msg='{msg_text[:50]}...'")
+                    # Fall through to text-based validation
+            else:
+                logger.warning(f"Message ID {message_id} not found in messages")
+                # Fall through to text-based validation
+        
+        # Strategy 2: Text-based validation (fallback)
+        # Normalize text for comparison
+        def normalize_text(text):
+            return text.lower().strip().replace('\n', ' ').replace('\r', '')
+        
+        normalized_quote = normalize_text(quote_text)
+        
+        # Look for the quote text in any message
+        for msg in messages:
+            msg_text = normalize_text(msg.get('text', ''))
+            
+            # Check for exact substring match
+            if normalized_quote in msg_text or msg_text in normalized_quote:
+                logger.debug(f"Quote validated via text matching in message: {msg.get('id', 'unknown')}")
+                return True
+            
+            # Check for high similarity (fuzzy matching)
+            if len(normalized_quote) > 10 and len(msg_text) > 10:
+                # Simple word overlap check
+                quote_words = set(normalized_quote.split())
+                msg_words = set(msg_text.split())
+                
+                if len(quote_words) > 0 and len(msg_words) > 0:
+                    overlap = len(quote_words.intersection(msg_words))
+                    quote_word_ratio = overlap / len(quote_words)
+                    msg_word_ratio = overlap / len(msg_words)
+                    
+                    # If 70% of quote words are in message, or 70% of message words are in quote
+                    if quote_word_ratio >= 0.7 or msg_word_ratio >= 0.7:
+                        logger.debug(f"Quote validated via fuzzy matching (overlap: {overlap}/{len(quote_words)} words)")
+                        return True
+        
+        logger.warning(f"Quote validation failed: no matching message found for '{quote_text[:50]}...'")
+        return False

backend/app/services/llm_service.py

@@ -0,0 +1,554 @@
+"""
+LLM Service for Synthetic Society
+This service provides a centralized interface for interacting with language models
+through the Google Generative AI API. It supports various prompting functions for
+different application features.
+"""
+
+import os
+import json
+import time
+import logging
+import google.generativeai as genai
+from typing import Dict, Any, Optional, Union, List
+from PIL import Image
+import io
+
+# Set up the Gemini API key
+GEMINI_API_KEY = os.environ.get('GEMINI_API_KEY', 'AIzaSyAc50jzC3k9K1PmKT1vGFi0sCdhhnqsvl0')
+genai.configure(api_key=GEMINI_API_KEY)
+
+# The default model we're using 
+DEFAULT_MODEL = "gemini-2.5-pro"
+
+class LLMServiceError(Exception):
+    """Exception raised for errors in LLM operations."""
+    pass
+
+class LLMService:
+    """Centralized service for LLM operations."""
+    
+    @staticmethod
+    def get_model(model_name: Optional[str] = None) -> genai.GenerativeModel:
+        """
+        Get a configured Gemini model.
+        
+        Args:
+            model_name: Optional model name to use. Defaults to the default model.
+            
+        Returns:
+            A configured Gemini generative model
+        """
+        return genai.GenerativeModel(model_name or DEFAULT_MODEL)
+    
+    @staticmethod
+    def _extract_text_from_response(response) -> str:
+        """
+        Extract text from a Gemini API response, handling both simple and multi-part responses.
+        
+        Args:
+            response: The response object from the Gemini API
+            
+        Returns:
+            The extracted text content
+            
+        Raises:
+            LLMServiceError: If no text content can be extracted
+        """
+        try:
+            # Try the simple text accessor first
+            return response.text.strip()
+        except Exception:
+            # If that fails, try to extract from parts using the recommended approach
+            try:
+                text_parts = []
+                
+                # Check if response has direct parts attribute (as suggested in error message)
+                if hasattr(response, 'parts') and response.parts:
+                    for part in response.parts:
+                        if hasattr(part, 'text'):
+                            text_parts.append(part.text)
+                
+                # If that didn't work, try the candidates approach
+                if not text_parts and hasattr(response, 'candidates') and response.candidates:
+                    for candidate in response.candidates:
+                        # Check if finish reason indicates blocking
+                        if candidate.finish_reason == 3:
+                            raise LLMServiceError("Response was blocked for safety reasons")
+                        elif candidate.finish_reason == 4:
+                            raise LLMServiceError("Response was blocked for recitation reasons")
+                        elif candidate.finish_reason == 2:
+                            raise LLMServiceError("Response was cut off due to length limit - try reducing max_tokens or removing the limit")
+                        
+                        if hasattr(candidate, 'content') and hasattr(candidate.content, 'parts'):
+                            for part in candidate.content.parts:
+                                if hasattr(part, 'text'):
+                                    text_parts.append(part.text)
+                
+                # Join all text parts if we found any
+                if text_parts:
+                    return ''.join(text_parts).strip()
+                
+                # If we still can't extract text, it might be a safety/blocking issue
+                if hasattr(response, 'candidates') and response.candidates:
+                    finish_reason = response.candidates[0].finish_reason
+                    if finish_reason == 3:
+                        raise LLMServiceError("Response was blocked for safety reasons")
+                    elif finish_reason == 4:
+                        raise LLMServiceError("Response was blocked for recitation reasons")
+                    elif finish_reason == 2:
+                        raise LLMServiceError("Response was cut off due to length limit - try reducing max_tokens or removing the limit")
+                
+                raise LLMServiceError("Unable to extract text from response parts")
+                
+            except Exception as e:
+                raise LLMServiceError(f"Error extracting text from multi-part response: {str(e)}")
+
+    @staticmethod
+    def generate_content(
+        prompt: str,
+        temperature: float = 0.7,
+        max_tokens: Optional[int] = None,
+        model_name: Optional[str] = None,
+        system_prompt: Optional[str] = None
+    ) -> str:
+        """
+        Generate content using the LLM with retry mechanism for transient errors.
+        
+        Args:
+            prompt: The prompt to send to the model
+            temperature: Controls randomness (0.0 = deterministic, 1.0 = creative)
+            max_tokens: Maximum number of tokens to generate
+            model_name: Optional model name to use
+            system_prompt: Optional system prompt to define the role of the AI
+            
+        Returns:
+            The generated text response
+            
+        Raises:
+            LLMServiceError: If there's an issue with the generation
+        """
+        logger = logging.getLogger(__name__)
+        max_retries = 3
+        last_error = None
+        
+        for attempt in range(max_retries):
+            attempt_num = attempt + 1
+            logger.debug(f"LLM content generation attempt {attempt_num}/{max_retries}")
+            
+            try:
+                model = LLMService.get_model(model_name)
+                
+                generation_config = {
+                    "temperature": temperature,
+                }
+                
+                if max_tokens:
+                    generation_config["max_output_tokens"] = max_tokens
+                
+                # If system prompt is provided, use it to create a structured chat
+                if system_prompt:
+                    # For Gemini models, system prompts need to be passed as part of the user prompt
+                    # as Gemini API doesn't support 'system' role directly
+                    response = model.generate_content(
+                        [
+                            {"role": "user", "parts": [f"System: {system_prompt}\n\nUser: {prompt}"]}
+                        ],
+                        generation_config=genai.types.GenerationConfig(**generation_config)
+                    )
+                else:
+                    # Otherwise use the standard prompt-only approach
+                    response = model.generate_content(
+                        prompt,
+                        generation_config=genai.types.GenerationConfig(**generation_config)
+                    )
+                
+                # If successful, extract and return the response
+                result = LLMService._extract_text_from_response(response)
+                
+                if attempt > 0:
+                    logger.info(f"LLM content generation succeeded on attempt {attempt_num}/{max_retries}")
+                return result
+                
+            except Exception as e:
+                last_error = e
+                error_message = str(e).lower()
+                
+                logger.warning(f"LLM attempt {attempt_num}/{max_retries} failed: {str(e)}")
+                
+                # Check if this is a retryable error (Google API internal errors, rate limiting, etc.)
+                if ("500" in error_message or 
+                    "internal error" in error_message or 
+                    "internal server error" in error_message or
+                    "service unavailable" in error_message or
+                    "timeout" in error_message or
+                    "rate" in error_message):
+                    
+                    if attempt < max_retries - 1:
+                        # Wait before retrying (exponential backoff)
+                        wait_time = 2 ** attempt  # 1s, 2s, 4s
+                        logger.info(f"Retryable error detected. Waiting {wait_time} seconds before retry {attempt_num + 1}/{max_retries}")
+                        time.sleep(wait_time)
+                        continue
+                    else:
+                        logger.error(f"Retryable error detected but max retries ({max_retries}) reached")
+                else:
+                    logger.error(f"Non-retryable error detected: {str(e)}")
+                    break
+        
+        # If we've exhausted all retries or hit a non-retryable error, raise the last error
+        logger.error(f"LLM content generation failed after {max_retries} attempts. Final error: {str(last_error)}")
+        raise LLMServiceError(f"Error generating content: {str(last_error)}")
+    
+    @staticmethod
+    def parse_json_response(response_text: str) -> Union[Dict[str, Any], List[Any]]:
+        """
+        Parse a JSON response from the LLM.
+        
+        Args:
+            response_text: The text response from the LLM
+            
+        Returns:
+            A dictionary or list parsed from the JSON response
+            
+        Raises:
+            LLMServiceError: If there's an issue parsing the JSON
+        """
+        try:
+            # Handle common formatting issues in the response
+            clean_response = response_text
+            
+            # Remove markdown code blocks if present
+            if clean_response.startswith("```json"):
+                clean_response = clean_response.strip("```json").strip("```").strip()
+            elif clean_response.startswith("```"):
+                clean_response = clean_response.strip("```").strip()
+            
+            # Parse the JSON
+            return json.loads(clean_response)
+            
+        except json.JSONDecodeError as e:
+            error_msg = f"Failed to parse JSON response: {str(e)}. Raw response: {response_text[:200]}..."
+            logger.error(error_msg)
+            raise LLMServiceError(error_msg)
+            
+    @staticmethod
+    def generate_structured_response(
+        prompt: str,
+        temperature: float = 0.7,
+        max_tokens: Optional[int] = None,
+        model_name: Optional[str] = None,
+        system_prompt: Optional[str] = None
+    ) -> Dict[str, Any]:
+        """
+        Generate a structured JSON response using the LLM.
+        
+        Args:
+            prompt: The prompt to send to the model
+            temperature: Controls randomness in generation
+            max_tokens: Maximum tokens to generate
+            model_name: Optional model name to use
+            system_prompt: Optional system prompt to define the role of the AI
+            
+        Returns:
+            A dictionary parsed from the JSON response
+            
+        Raises:
+            LLMServiceError: If there's an issue with generation or parsing
+        """
+        response_text = LLMService.generate_content(
+            prompt=prompt,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            model_name=model_name,
+            system_prompt=system_prompt
+        )
+        
+        return LLMService.parse_json_response(response_text)
+    
+    @staticmethod
+    def generate_structured_array(
+        prompt: str,
+        temperature: float = 0.7,
+        max_tokens: Optional[int] = None,
+        model_name: Optional[str] = None,
+        system_prompt: Optional[str] = None
+    ) -> List[Dict[str, Any]]:
+        """
+        Generate a structured JSON array response using the LLM.
+        
+        Args:
+            prompt: The prompt to send to the model
+            temperature: Controls randomness in generation
+            max_tokens: Maximum tokens to generate
+            model_name: Optional model name to use
+            system_prompt: Optional system prompt to define the role of the AI
+            
+        Returns:
+            A list of dictionaries parsed from the JSON array response
+            
+        Raises:
+            LLMServiceError: If there's an issue with generation or parsing
+        """
+        response_text = LLMService.generate_content(
+            prompt=prompt,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            model_name=model_name,
+            system_prompt=system_prompt
+        )
+        
+        result = LLMService.parse_json_response(response_text)
+        
+        # Ensure the result is a list
+        if not isinstance(result, list):
+            raise LLMServiceError(f"Expected a JSON array but received {type(result)}")
+            
+        return result
+    
+    @staticmethod
+    def generate_multimodal_content(
+        prompt: str,
+        image_paths: List[str],
+        temperature: float = 0.7,
+        max_tokens: Optional[int] = None,
+        model_name: Optional[str] = None
+    ) -> str:
+        """
+        Generate content using both text and image inputs.
+        
+        Args:
+            prompt: The text prompt to send to the model
+            image_paths: List of paths to image files to include
+            temperature: Controls randomness in generation
+            max_tokens: Maximum tokens to generate
+            model_name: Optional model name to use
+            
+        Returns:
+            The generated text response
+            
+        Raises:
+            LLMServiceError: If there's an issue with generation or image processing
+        """
+        logger = logging.getLogger(__name__)
+        max_retries = 3
+        last_error = None
+        
+        # Load and validate images
+        images = []
+        for image_path in image_paths:
+            try:
+                if not os.path.exists(image_path):
+                    raise LLMServiceError(f"Image file not found: {image_path}")
+                
+                # Load image using PIL
+                with Image.open(image_path) as img:
+                    # Convert to RGB if necessary
+                    if img.mode != 'RGB':
+                        img = img.convert('RGB')
+                    images.append(img.copy())
+                    
+                logger.debug(f"Successfully loaded image: {image_path}")
+                
+            except Exception as e:
+                raise LLMServiceError(f"Failed to load image {image_path}: {str(e)}")
+        
+        logger.info(f"Generating multimodal content with {len(images)} image(s)")
+        
+        for attempt in range(max_retries):
+            attempt_num = attempt + 1
+            logger.debug(f"Multimodal content generation attempt {attempt_num}/{max_retries}")
+            
+            try:
+                model = LLMService.get_model(model_name)
+                
+                generation_config = {
+                    "temperature": temperature,
+                }
+                
+                if max_tokens:
+                    generation_config["max_output_tokens"] = max_tokens
+                
+                # Create multimodal input - combine text prompt with images
+                content_parts = [prompt]
+                content_parts.extend(images)
+            
+                response = model.generate_content(
+                    content_parts,
+                    generation_config=genai.types.GenerationConfig(**generation_config)
+                )
+                
+                # Extract and return the response
+                result = LLMService._extract_text_from_response(response)
+                
+                if attempt > 0:
+                    logger.info(f"Multimodal content generation succeeded on attempt {attempt_num}/{max_retries}")
+                return result
+                
+            except Exception as e:
+                last_error = e
+                error_message = str(e).lower()
+                
+                logger.warning(f"Multimodal attempt {attempt_num}/{max_retries} failed: {str(e)}")
+                
+                # Check if this is a retryable error
+                if ("500" in error_message or 
+                    "internal error" in error_message or 
+                    "internal server error" in error_message or
+                    "service unavailable" in error_message or
+                    "timeout" in error_message or
+                    "rate" in error_message):
+                    
+                    if attempt < max_retries - 1:
+                        # Wait before retrying (exponential backoff)
+                        wait_time = 2 ** attempt  # 1s, 2s, 4s
+                        logger.info(f"Retryable error detected. Waiting {wait_time} seconds before retry {attempt_num + 1}/{max_retries}")
+                        time.sleep(wait_time)
+                        continue
+                    else:
+                        logger.error(f"Retryable error detected but max retries ({max_retries}) reached")
+                else:
+                    logger.error(f"Non-retryable error detected: {str(e)}")
+                    break
+        
+        # If we've exhausted all retries or hit a non-retryable error, raise the last error
+        logger.error(f"Multimodal content generation failed after {max_retries} attempts. Final error: {str(last_error)}")
+        raise LLMServiceError(f"Error generating multimodal content: {str(last_error)}")
+    
+    @staticmethod
+    def generate_contextual_response(
+        prompt: str,
+        conversation_context: List[Dict[str, Any]],
+        temperature: float = 0.7,
+        max_tokens: Optional[int] = None,
+        model_name: Optional[str] = None
+    ) -> str:
+        """
+        Generate content using conversation context that may include both text and images in sequence.
+        
+        Args:
+            prompt: The main prompt for the LLM
+            conversation_context: List of context items (text and image) in chronological order
+            temperature: Controls randomness in generation
+            max_tokens: Maximum tokens to generate
+            model_name: Optional model name to use
+            
+        Returns:
+            The generated text response
+            
+        Raises:
+            LLMServiceError: If there's an issue with generation
+        """
+        logger = logging.getLogger(__name__)
+        
+        # Separate text and image content from the conversation context
+        text_context_parts = []
+        image_parts = []
+        
+        print(f"🎯 Processing {len(conversation_context)} context items for LLM")
+        
+        for item in conversation_context:
+            if item["type"] == "text":
+                text_context_parts.append(item["content"])
+            elif item["type"] == "image":
+                try:
+                    image_path = item["path"]
+                    if os.path.exists(image_path):
+                        # Load image using PIL
+                        with Image.open(image_path) as img:
+                            # Convert to RGB if necessary
+                            if img.mode != 'RGB':
+                                img = img.convert('RGB')
+                            image_parts.append(img.copy())
+                            print(f"🖼️  Loaded image for context: {item['filename']}")
+                    else:
+                        print(f"⚠️  Image not found for context: {image_path}")
+                except Exception as e:
+                    print(f"❌ Failed to load image for context: {item['path']}: {e}")
+        
+        # Build the full context prompt
+        context_prompt = ""
+        if text_context_parts:
+            context_prompt = "CONVERSATION CONTEXT:\n" + "\n".join(text_context_parts) + "\n\n"
+        
+        full_prompt = context_prompt + prompt
+        
+        print(f"📝 Context prompt length: {len(context_prompt)} characters")
+        print(f"🖼️  Total images in context: {len(image_parts)}")
+        
+        # If we have images, use multimodal generation
+        if image_parts:
+            print(f"🎨 Using multimodal generation with {len(image_parts)} images")
+            
+            # Create content parts with text and images
+            content_parts = [full_prompt]
+            content_parts.extend(image_parts)
+            
+            max_retries = 3
+            last_error = None
+            
+            for attempt in range(max_retries):
+                attempt_num = attempt + 1
+                logger.debug(f"Contextual multimodal generation attempt {attempt_num}/{max_retries}")
+                
+                try:
+                    model = LLMService.get_model(model_name)
+                    
+                    generation_config = {
+                        "temperature": temperature,
+                    }
+                    
+                    if max_tokens:
+                        generation_config["max_output_tokens"] = max_tokens
+                    
+                    response = model.generate_content(
+                        content_parts,
+                        generation_config=genai.types.GenerationConfig(**generation_config)
+                    )
+                    
+                    result = LLMService._extract_text_from_response(response)
+                    
+                    if attempt > 0:
+                        logger.info(f"Contextual multimodal generation succeeded on attempt {attempt_num}/{max_retries}")
+                    
+                    print(f"✅ Generated contextual response with visual context")
+                    return result
+                    
+                except Exception as e:
+                    last_error = e
+                    error_message = str(e).lower()
+                    
+                    logger.warning(f"Contextual multimodal attempt {attempt_num}/{max_retries} failed: {str(e)}")
+                    
+                    # Check if this is a retryable error
+                    if ("500" in error_message or 
+                        "internal error" in error_message or 
+                        "internal server error" in error_message or
+                        "service unavailable" in error_message or
+                        "timeout" in error_message or
+                        "rate" in error_message):
+                        
+                        if attempt < max_retries - 1:
+                            wait_time = 2 ** attempt
+                            logger.info(f"Retryable error detected. Waiting {wait_time} seconds before retry {attempt_num + 1}/{max_retries}")
+                            time.sleep(wait_time)
+                            continue
+                        else:
+                            logger.error(f"Retryable error detected but max retries ({max_retries}) reached")
+                    else:
+                        logger.error(f"Non-retryable error detected: {str(e)}")
+                        break
+            
+            # If multimodal failed, raise the error
+            logger.error(f"Contextual multimodal generation failed after {max_retries} attempts. Final error: {str(last_error)}")
+            raise LLMServiceError(f"Error generating contextual multimodal content: {str(last_error)}")
+        
+        else:
+            # No images, use standard text generation
+            print(f"📝 Using text-only generation (no visual context)")
+            return LLMService.generate_content(
+                prompt=full_prompt,
+                temperature=temperature,
+                max_tokens=max_tokens,
+                model_name=model_name
+            )

backend/app/utils.py

@@ -0,0 +1,17 @@
+from functools import wraps
+from flask import jsonify
+from flask_jwt_extended import get_jwt_identity
+from app.models.user import User
+
+def admin_required(f):
+    @wraps(f)
+    def decorated_function(*args, **kwargs):
+        user_id = get_jwt_identity()
+        user_data = User.find_by_id(user_id)
+        
+        if not user_data or user_data.get('role') != 'admin':
+            return jsonify({"message": "Admin privileges required"}), 403
+        
+        return f(*args, **kwargs)
+    
+    return decorated_function

backend/app/utils/__init__.py

@@ -0,0 +1 @@
+# Utils package for Synthetic Society backend

backend/app/utils/discussion_guide_schema.py

@@ -0,0 +1,422 @@
+"""
+Discussion Guide JSON Schema Validation
+Provides schema validation and utilities for structured discussion guides.
+"""
+
+from typing import Dict, List, Any, Optional, Union
+from dataclasses import dataclass
+import json
+
+
+@dataclass
+class DiscussionGuideActivity:
+    """Represents an activity within a discussion guide section."""
+    id: str
+    type: str  # moderator_statement, open_question, probe_question, activity, etc.
+    content: str
+    time_limit: Optional[int] = None
+    metadata: Optional[Dict[str, Any]] = None
+
+
+@dataclass
+class DiscussionGuideQuestion:
+    """Represents a question within a discussion guide section."""
+    id: str
+    type: str  # open_question, probe_question, follow_up, etc.
+    content: str
+    time_limit: Optional[int] = None
+    probes: Optional[List[str]] = None
+    metadata: Optional[Dict[str, Any]] = None
+
+
+@dataclass
+class DiscussionGuideSubsection:
+    """Represents a subsection within a main discussion section."""
+    id: str
+    title: str
+    duration: int
+    questions: List[DiscussionGuideQuestion]
+    activities: Optional[List[DiscussionGuideActivity]] = None
+    metadata: Optional[Dict[str, Any]] = None
+
+
+@dataclass
+class DiscussionGuideSection:
+    """Represents a main section of a discussion guide."""
+    id: str
+    title: str
+    duration: int
+    type: str  # introduction, warmup, main_content, conclusion
+    content: Optional[str] = None
+    questions: Optional[List[DiscussionGuideQuestion]] = None
+    activities: Optional[List[DiscussionGuideActivity]] = None
+    subsections: Optional[List[DiscussionGuideSubsection]] = None
+    metadata: Optional[Dict[str, Any]] = None
+
+
+@dataclass
+class StructuredDiscussionGuide:
+    """Represents a complete structured discussion guide."""
+    title: str
+    total_duration: int
+    sections: List[DiscussionGuideSection]
+    metadata: Optional[Dict[str, Any]] = None
+
+
+class DiscussionGuideValidator:
+    """Validates and processes discussion guide JSON structures."""
+    
+    @staticmethod
+    def validate_json_structure(guide_json: Dict[str, Any]) -> tuple[bool, List[str]]:
+        """
+        Validate a discussion guide JSON structure.
+        
+        Args:
+            guide_json: The JSON structure to validate
+            
+        Returns:
+            Tuple of (is_valid, list_of_errors)
+        """
+        errors = []
+        
+        # Check required top-level fields
+        required_fields = ['title', 'total_duration', 'sections']
+        for field in required_fields:
+            if field not in guide_json:
+                errors.append(f"Missing required field: {field}")
+        
+        if 'sections' in guide_json:
+            if not isinstance(guide_json['sections'], list):
+                errors.append("'sections' must be a list")
+            elif len(guide_json['sections']) == 0:
+                errors.append("'sections' cannot be empty")
+            else:
+                # Validate each section
+                for i, section in enumerate(guide_json['sections']):
+                    section_errors = DiscussionGuideValidator._validate_section(section, i)
+                    errors.extend(section_errors)
+        
+        # Validate total duration matches sum of sections
+        if 'sections' in guide_json and 'total_duration' in guide_json:
+            try:
+                total_section_duration = sum(section.get('duration', 0) for section in guide_json['sections'])
+                if abs(total_section_duration - guide_json['total_duration']) > 5:  # Allow 5 minute tolerance
+                    errors.append(f"Total duration ({guide_json['total_duration']}) doesn't match sum of sections ({total_section_duration})")
+            except (TypeError, ValueError):
+                errors.append("Invalid duration values in sections")
+        
+        return len(errors) == 0, errors
+    
+    @staticmethod
+    def _validate_section(section: Dict[str, Any], index: int) -> List[str]:
+        """Validate a single section."""
+        errors = []
+        section_prefix = f"Section {index + 1}"
+        
+        # Check required section fields
+        required_fields = ['id', 'title', 'duration', 'type']
+        for field in required_fields:
+            if field not in section:
+                errors.append(f"{section_prefix}: Missing required field '{field}'")
+        
+        # Validate section type
+        if 'type' in section:
+            valid_types = ['introduction', 'warmup', 'main_content', 'conclusion', 'activity', 'break']
+            if section['type'] not in valid_types:
+                errors.append(f"{section_prefix}: Invalid section type '{section['type']}'")
+        
+        # Validate duration
+        if 'duration' in section:
+            try:
+                duration = int(section['duration'])
+                if duration <= 0:
+                    errors.append(f"{section_prefix}: Duration must be positive")
+            except (TypeError, ValueError):
+                errors.append(f"{section_prefix}: Duration must be a number")
+        
+        # Validate questions if present
+        if 'questions' in section and section['questions']:
+            if not isinstance(section['questions'], list):
+                errors.append(f"{section_prefix}: 'questions' must be a list")
+            else:
+                for j, question in enumerate(section['questions']):
+                    question_errors = DiscussionGuideValidator._validate_question(question, index, j)
+                    errors.extend(question_errors)
+        
+        # Validate activities if present
+        if 'activities' in section and section['activities']:
+            if not isinstance(section['activities'], list):
+                errors.append(f"{section_prefix}: 'activities' must be a list")
+            else:
+                for j, activity in enumerate(section['activities']):
+                    activity_errors = DiscussionGuideValidator._validate_activity(activity, index, j)
+                    errors.extend(activity_errors)
+        
+        # Validate subsections if present
+        if 'subsections' in section and section['subsections']:
+            if not isinstance(section['subsections'], list):
+                errors.append(f"{section_prefix}: 'subsections' must be a list")
+            else:
+                for j, subsection in enumerate(section['subsections']):
+                    subsection_errors = DiscussionGuideValidator._validate_subsection(subsection, index, j)
+                    errors.extend(subsection_errors)
+        
+        return errors
+    
+    @staticmethod
+    def _validate_question(question: Dict[str, Any], section_index: int, question_index: int) -> List[str]:
+        """Validate a single question."""
+        errors = []
+        question_prefix = f"Section {section_index + 1}, Question {question_index + 1}"
+        
+        # Check required question fields
+        required_fields = ['id', 'type', 'content']
+        for field in required_fields:
+            if field not in question:
+                errors.append(f"{question_prefix}: Missing required field '{field}'")
+        
+        # Validate question type (accept any string)
+        if 'type' in question and not isinstance(question['type'], str):
+            errors.append(f"{question_prefix}: Question type must be a string")
+        
+        # Validate content
+        if 'content' in question and not isinstance(question['content'], str):
+            errors.append(f"{question_prefix}: 'content' must be a string")
+        
+        return errors
+    
+    @staticmethod
+    def _validate_activity(activity: Dict[str, Any], section_index: int, activity_index: int) -> List[str]:
+        """Validate a single activity."""
+        errors = []
+        activity_prefix = f"Section {section_index + 1}, Activity {activity_index + 1}"
+        
+        # Check required activity fields
+        required_fields = ['id', 'type', 'content']
+        for field in required_fields:
+            if field not in activity:
+                errors.append(f"{activity_prefix}: Missing required field '{field}'")
+        
+        # Validate activity type (accept any string)
+        if 'type' in activity and not isinstance(activity['type'], str):
+            errors.append(f"{activity_prefix}: Activity type must be a string")
+        
+        return errors
+    
+    @staticmethod
+    def _validate_subsection(subsection: Dict[str, Any], section_index: int, subsection_index: int) -> List[str]:
+        """Validate a single subsection."""
+        errors = []
+        subsection_prefix = f"Section {section_index + 1}, Subsection {subsection_index + 1}"
+        
+        # Check required subsection fields
+        required_fields = ['id', 'title', 'duration', 'questions']
+        for field in required_fields:
+            if field not in subsection:
+                errors.append(f"{subsection_prefix}: Missing required field '{field}'")
+        
+        # Validate questions
+        if 'questions' in subsection and subsection['questions']:
+            if not isinstance(subsection['questions'], list):
+                errors.append(f"{subsection_prefix}: 'questions' must be a list")
+            else:
+                for j, question in enumerate(subsection['questions']):
+                    question_errors = DiscussionGuideValidator._validate_question(question, section_index, j)
+                    errors.extend(question_errors)
+        
+        return errors
+    
+    @staticmethod
+    def create_fallback_structure(title: str, duration: int, content: str) -> Dict[str, Any]:
+        """
+        Create a fallback discussion guide structure if JSON generation fails.
+        
+        Args:
+            title: The focus group title
+            duration: Total duration in minutes
+            content: Raw content to structure
+            
+        Returns:
+            A basic valid discussion guide structure
+        """
+        # Calculate section durations
+        intro_duration = max(5, int(duration * 0.1))
+        warmup_duration = max(5, int(duration * 0.15))
+        main_duration = max(20, int(duration * 0.6))
+        conclusion_duration = max(5, int(duration * 0.15))
+        
+        return {
+            "title": title,
+            "total_duration": duration,
+            "sections": [
+                {
+                    "id": "introduction",
+                    "title": "Introduction",
+                    "duration": intro_duration,
+                    "type": "introduction",
+                    "activities": [
+                        {
+                            "id": "welcome",
+                            "type": "moderator_statement",
+                            "content": f"Welcome everyone to our focus group on {title}. Let's begin by introducing ourselves and the purpose of today's discussion."
+                        }
+                    ]
+                },
+                {
+                    "id": "warmup",
+                    "title": "Warm-up Questions",
+                    "duration": warmup_duration,
+                    "type": "warmup",
+                    "questions": [
+                        {
+                            "id": "intro_question",
+                            "type": "open_question",
+                            "content": "Let's start with brief introductions. Please share your name and one thing you're excited about today."
+                        }
+                    ]
+                },
+                {
+                    "id": "main_discussion",
+                    "title": "Main Discussion",
+                    "duration": main_duration,
+                    "type": "main_content",
+                    "questions": [
+                        {
+                            "id": "main_question",
+                            "type": "open_question",
+                            "content": "What are your initial thoughts on the topic we're discussing today?"
+                        }
+                    ]
+                },
+                {
+                    "id": "conclusion",
+                    "title": "Conclusion",
+                    "duration": conclusion_duration,
+                    "type": "conclusion",
+                    "questions": [
+                        {
+                            "id": "final_thoughts",
+                            "type": "open_question",
+                            "content": "Before we wrap up, are there any final thoughts or comments you'd like to share?"
+                        }
+                    ]
+                }
+            ]
+        }
+    
+    @staticmethod
+    def parse_from_json(json_string: str) -> tuple[Optional[StructuredDiscussionGuide], List[str]]:
+        """
+        Parse a JSON string into a StructuredDiscussionGuide object.
+        
+        Args:
+            json_string: The JSON string to parse
+            
+        Returns:
+            Tuple of (parsed_guide, list_of_errors)
+        """
+        try:
+            guide_json = json.loads(json_string)
+        except json.JSONDecodeError as e:
+            return None, [f"Invalid JSON: {str(e)}"]
+        
+        # Validate structure
+        is_valid, errors = DiscussionGuideValidator.validate_json_structure(guide_json)
+        if not is_valid:
+            return None, errors
+        
+        try:
+            # Convert to structured objects
+            sections = []
+            for section_data in guide_json['sections']:
+                section = DiscussionGuideValidator._parse_section(section_data)
+                sections.append(section)
+            
+            guide = StructuredDiscussionGuide(
+                title=guide_json['title'],
+                total_duration=guide_json['total_duration'],
+                sections=sections,
+                metadata=guide_json.get('metadata')
+            )
+            
+            return guide, []
+            
+        except Exception as e:
+            return None, [f"Error parsing structure: {str(e)}"]
+    
+    @staticmethod
+    def _parse_section(section_data: Dict[str, Any]) -> DiscussionGuideSection:
+        """Parse a section from JSON data."""
+        questions = []
+        if section_data.get('questions'):
+            for q_data in section_data['questions']:
+                question = DiscussionGuideQuestion(
+                    id=q_data['id'],
+                    type=q_data['type'],
+                    content=q_data['content'],
+                    time_limit=q_data.get('time_limit'),
+                    probes=q_data.get('probes'),
+                    metadata=q_data.get('metadata')
+                )
+                questions.append(question)
+        
+        activities = []
+        if section_data.get('activities'):
+            for a_data in section_data['activities']:
+                activity = DiscussionGuideActivity(
+                    id=a_data['id'],
+                    type=a_data['type'],
+                    content=a_data['content'],
+                    time_limit=a_data.get('time_limit'),
+                    metadata=a_data.get('metadata')
+                )
+                activities.append(activity)
+        
+        subsections = []
+        if section_data.get('subsections'):
+            for s_data in section_data['subsections']:
+                subsection_questions = []
+                for q_data in s_data.get('questions', []):
+                    question = DiscussionGuideQuestion(
+                        id=q_data['id'],
+                        type=q_data['type'],
+                        content=q_data['content'],
+                        time_limit=q_data.get('time_limit'),
+                        probes=q_data.get('probes'),
+                        metadata=q_data.get('metadata')
+                    )
+                    subsection_questions.append(question)
+                
+                subsection_activities = []
+                if s_data.get('activities'):
+                    for a_data in s_data['activities']:
+                        activity = DiscussionGuideActivity(
+                            id=a_data['id'],
+                            type=a_data['type'],
+                            content=a_data['content'],
+                            time_limit=a_data.get('time_limit'),
+                            metadata=a_data.get('metadata')
+                        )
+                        subsection_activities.append(activity)
+                
+                subsection = DiscussionGuideSubsection(
+                    id=s_data['id'],
+                    title=s_data['title'],
+                    duration=s_data['duration'],
+                    questions=subsection_questions,
+                    activities=subsection_activities if subsection_activities else None,
+                    metadata=s_data.get('metadata')
+                )
+                subsections.append(subsection)
+        
+        return DiscussionGuideSection(
+            id=section_data['id'],
+            title=section_data['title'],
+            duration=section_data['duration'],
+            type=section_data['type'],
+            content=section_data.get('content'),
+            questions=questions if questions else None,
+            activities=activities if activities else None,
+            subsections=subsections if subsections else None,
+            metadata=section_data.get('metadata')
+        )

backend/app/utils/prompt_loader.py

@@ -0,0 +1,207 @@
+"""
+Prompt Loading Utility for Synthetic Society
+This utility provides centralized prompt loading from markdown files
+with template variable substitution support.
+"""
+
+import os
+from typing import Dict, Any, Optional
+from pathlib import Path
+
+class PromptLoaderError(Exception):
+    """Exception raised for errors in prompt loading."""
+    pass
+
+class PromptLoader:
+    """Centralized prompt loading service."""
+    
+    def __init__(self, prompts_dir: Optional[str] = None):
+        """
+        Initialize the prompt loader.
+        
+        Args:
+            prompts_dir: Optional directory path for prompts. Defaults to 
+                        backend/prompts relative to this file's location.
+        """
+        if prompts_dir:
+            self.prompts_dir = Path(prompts_dir)
+        else:
+            # Default to prompts directory in backend folder
+            current_file = Path(__file__)
+            backend_dir = current_file.parent.parent.parent  # Go up from app/utils to backend
+            self.prompts_dir = backend_dir / "prompts"
+        
+        if not self.prompts_dir.exists():
+            raise PromptLoaderError(f"Prompts directory not found: {self.prompts_dir}")
+    
+    def load_prompt(self, prompt_name: str, variables: Optional[Dict[str, Any]] = None) -> str:
+        """
+        Load a prompt from a markdown file and substitute template variables.
+        
+        Args:
+            prompt_name: Name of the prompt file (without .md extension)
+            variables: Dictionary of variables to substitute in the template
+            
+        Returns:
+            The prompt text with variables substituted
+            
+        Raises:
+            PromptLoaderError: If the prompt file is not found or cannot be processed
+        """
+        try:
+            prompt_file = self.prompts_dir / f"{prompt_name}.md"
+            
+            if not prompt_file.exists():
+                raise PromptLoaderError(f"Prompt file not found: {prompt_file}")
+            
+            # Read the prompt content
+            with open(prompt_file, 'r', encoding='utf-8') as f:
+                prompt_content = f.read().strip()
+            
+            # Substitute variables if provided
+            if variables:
+                try:
+                    # Handle JSON examples by protecting them from template substitution
+                    # Look for JSON example blocks and temporarily replace them
+                    json_blocks = []
+                    import re
+                    
+                    # Find JSON example blocks (between EXAMPLE_JSON_START and EXAMPLE_JSON_END)
+                    json_pattern = r'EXAMPLE_JSON_START\s*\n(.*?)\nEXAMPLE_JSON_END'
+                    matches = list(re.finditer(json_pattern, prompt_content, re.DOTALL))
+                    
+                    # Replace in reverse order to avoid index shifting issues
+                    for i, match in enumerate(reversed(matches)):
+                        json_blocks.insert(0, match.group(1))
+                        prompt_content = prompt_content[:match.start()] + f"__JSON_PLACEHOLDER_{len(matches)-1-i}__" + prompt_content[match.end():]
+                    
+                    # Also protect any remaining JSON-like patterns that might have been missed
+                    # This is a fallback for any {key: value} patterns
+                    json_fallback_pattern = r'\{[^{}]*"[^"]*"[^{}]*\}'
+                    fallback_matches = list(re.finditer(json_fallback_pattern, prompt_content))
+                    
+                    for i, match in enumerate(reversed(fallback_matches)):
+                        json_blocks.insert(0, match.group(0))
+                        prompt_content = prompt_content[:match.start()] + f"__JSON_FALLBACK_{len(fallback_matches)-1-i}__" + prompt_content[match.end():]
+                    
+                    # Now do the template substitution
+                    prompt_content = prompt_content.format(**variables)
+                    
+                    # Restore the JSON blocks
+                    for i, json_block in enumerate(json_blocks[:len(matches)]):
+                        prompt_content = prompt_content.replace(f"__JSON_PLACEHOLDER_{i}__", json_block)
+                    
+                    # Restore fallback blocks
+                    for i, json_block in enumerate(json_blocks[len(matches):]):
+                        fallback_index = len(json_blocks[len(matches):]) - 1 - i
+                        prompt_content = prompt_content.replace(f"__JSON_FALLBACK_{fallback_index}__", json_block)
+                        
+                except KeyError as e:
+                    raise PromptLoaderError(f"Missing template variable in prompt '{prompt_name}': {e}")
+                except ValueError as e:
+                    raise PromptLoaderError(f"Template formatting error in prompt '{prompt_name}': {e}")
+            
+            return prompt_content
+            
+        except Exception as e:
+            if isinstance(e, PromptLoaderError):
+                raise
+            raise PromptLoaderError(f"Error loading prompt '{prompt_name}': {str(e)}")
+    
+    def list_available_prompts(self) -> list:
+        """
+        List all available prompt files.
+        
+        Returns:
+            List of prompt names (without .md extension)
+        """
+        try:
+            prompt_files = []
+            for file_path in self.prompts_dir.glob("*.md"):
+                prompt_files.append(file_path.stem)
+            return sorted(prompt_files)
+        except Exception as e:
+            raise PromptLoaderError(f"Error listing prompts: {str(e)}")
+    
+    def validate_prompt_variables(self, prompt_name: str, variables: Dict[str, Any]) -> bool:
+        """
+        Validate that all required variables are provided for a prompt.
+        
+        Args:
+            prompt_name: Name of the prompt file
+            variables: Dictionary of variables to validate
+            
+        Returns:
+            True if all required variables are present
+            
+        Raises:
+            PromptLoaderError: If validation fails
+        """
+        try:
+            # Load the raw prompt to check for template variables
+            prompt_file = self.prompts_dir / f"{prompt_name}.md"
+            
+            if not prompt_file.exists():
+                raise PromptLoaderError(f"Prompt file not found: {prompt_file}")
+            
+            with open(prompt_file, 'r', encoding='utf-8') as f:
+                prompt_content = f.read()
+            
+            # Try to format with provided variables to see if any are missing
+            try:
+                # Handle JSON examples by protecting them from template substitution
+                import re
+                json_blocks = []
+                json_pattern = r'EXAMPLE_JSON_START\s*\n(.*?)\nEXAMPLE_JSON_END'
+                matches = list(re.finditer(json_pattern, prompt_content, re.DOTALL))
+                
+                # Replace in reverse order to avoid index shifting issues
+                for i, match in enumerate(reversed(matches)):
+                    json_blocks.insert(0, match.group(1))
+                    prompt_content = prompt_content[:match.start()] + f"__JSON_PLACEHOLDER_{len(matches)-1-i}__" + prompt_content[match.end():]
+                
+                # Also protect any remaining JSON-like patterns
+                json_fallback_pattern = r'\{[^{}]*"[^"]*"[^{}]*\}'
+                fallback_matches = list(re.finditer(json_fallback_pattern, prompt_content))
+                
+                for i, match in enumerate(reversed(fallback_matches)):
+                    json_blocks.insert(0, match.group(0))
+                    prompt_content = prompt_content[:match.start()] + f"__JSON_FALLBACK_{len(fallback_matches)-1-i}__" + prompt_content[match.end():]
+                
+                prompt_content.format(**variables)
+                return True
+            except KeyError as e:
+                raise PromptLoaderError(f"Missing required variable for prompt '{prompt_name}': {e}")
+                
+        except Exception as e:
+            if isinstance(e, PromptLoaderError):
+                raise
+            raise PromptLoaderError(f"Error validating prompt variables: {str(e)}")
+
+# Global prompt loader instance
+_prompt_loader = None
+
+def get_prompt_loader() -> PromptLoader:
+    """
+    Get the global prompt loader instance.
+    
+    Returns:
+        The global PromptLoader instance
+    """
+    global _prompt_loader
+    if _prompt_loader is None:
+        _prompt_loader = PromptLoader()
+    return _prompt_loader
+
+def load_prompt(prompt_name: str, variables: Optional[Dict[str, Any]] = None) -> str:
+    """
+    Convenience function to load a prompt using the global loader.
+    
+    Args:
+        prompt_name: Name of the prompt file (without .md extension)
+        variables: Dictionary of variables to substitute in the template
+        
+    Returns:
+        The prompt text with variables substituted
+    """
+    return get_prompt_loader().load_prompt(prompt_name, variables)

backend/hypercorn_config.py

@@ -0,0 +1,25 @@
+"""
+Hypercorn configuration file for production deployment.
+
+This sets longer timeouts for AI generation requests.
+"""
+
+# Bind to 0.0.0.0:5137
+bind = "0.0.0.0:5137"
+
+# Worker configuration
+workers = 4
+worker_class = "asyncio"
+
+# Connection settings
+keep_alive = 65
+max_requests = 1000
+max_requests_jitter = 50
+
+# Logging
+accesslog = None  # Disable access logging for successful requests
+errorlog = "-"   # Log to stderr
+log_level = "info"  # Keep info level for application logs
+
+# Application settings
+application_path = "run:app"

backend/logging_config.py

@@ -0,0 +1,101 @@
+"""
+Custom logging configuration for the application to reduce noise.
+"""
+
+import logging
+import sys
+from typing import Dict, Any
+
+class CustomHTTPFilter(logging.Filter):
+    """Filter to reduce noise from routine HTTP requests."""
+    
+    # Define routes that should be logged only on errors/warnings
+    QUIET_ROUTES = {
+        '/api/focus-groups/',
+        '/api/focus-group-ai/key-themes/',
+        '/api/focus-group-ai/autonomous/status/',
+        '/api/personas',
+        '/api/ai-personas',
+        '/static/',
+        '/favicon.ico',
+        '/health',
+        '/ping'
+    }
+    
+    # Define HTTP methods that should be quiet for routine operations
+    QUIET_METHODS = {'GET', 'OPTIONS'}
+    
+    def filter(self, record: logging.LogRecord) -> bool:
+        """Filter log records to reduce noise from routine operations."""
+        
+        # Allow all non-HTTP logs
+        if not hasattr(record, 'pathname') and not hasattr(record, 'msg'):
+            return True
+            
+        # Check if this is an HTTP access log
+        msg = str(record.msg) if hasattr(record, 'msg') else str(record)
+        
+        # If it's not an HTTP request log, allow it
+        if not any(method in msg for method in ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS']):
+            return True
+        
+        # Check if it's a successful response (2xx) to a quiet route
+        if any(route in msg for route in self.QUIET_ROUTES):
+            # Allow if it's an error response (4xx, 5xx) or warning/error level
+            if record.levelno >= logging.WARNING:
+                return True
+            # Check if it's a successful response (2xx)
+            if ' 2' in msg and any(method in msg for method in self.QUIET_METHODS):
+                return False  # Filter out successful GET requests to quiet routes
+        
+        return True
+
+def setup_logging(log_level: str = 'INFO') -> None:
+    """Setup custom logging configuration."""
+    
+    # Convert log level string to logging constant
+    numeric_level = getattr(logging, log_level.upper(), logging.INFO)
+    
+    # Create custom formatter
+    formatter = logging.Formatter(
+        '[%(asctime)s] %(levelname)s in %(name)s: %(message)s',
+        datefmt='%Y-%m-%d %H:%M:%S'
+    )
+    
+    # Setup root logger
+    root_logger = logging.getLogger()
+    root_logger.setLevel(numeric_level)
+    
+    # Clear existing handlers
+    root_logger.handlers.clear()
+    
+    # Create console handler
+    console_handler = logging.StreamHandler(sys.stdout)
+    console_handler.setFormatter(formatter)
+    console_handler.setLevel(numeric_level)
+    
+    # Add custom filter to reduce HTTP noise
+    http_filter = CustomHTTPFilter()
+    console_handler.addFilter(http_filter)
+    
+    root_logger.addHandler(console_handler)
+    
+    # Set specific logger levels
+    logging.getLogger('pymongo').setLevel(logging.WARNING)  # Reduce MongoDB driver noise
+    logging.getLogger('werkzeug').setLevel(logging.WARNING)  # Reduce Flask dev server noise
+    logging.getLogger('hypercorn').setLevel(logging.WARNING)  # Reduce Hypercorn noise
+    logging.getLogger('hypercorn.access').setLevel(logging.WARNING)  # Reduce access log noise
+    
+    # Keep application loggers at INFO level
+    logging.getLogger('app').setLevel(numeric_level)
+    
+    # Setup MongoDB connection logging to be quieter
+    logging.getLogger('app.db').setLevel(logging.WARNING)
+
+def create_app_logger(name: str) -> logging.Logger:
+    """Create a logger for application modules."""
+    logger = logging.getLogger(f'app.{name}')
+    return logger
+
+# Default configuration - changed to DEBUG for troubleshooting
+DEFAULT_LOG_LEVEL = 'INFO'

backend/prompts/ai-moderator-system.md

@@ -0,0 +1,56 @@
+You are an expert focus group moderator conducting a professional research session. Your role is to facilitate the discussion by following the structured discussion guide precisely while maintaining a natural, conversational flow.
+
+## CURRENT CONTEXT:
+- **Section**: {section_title} ({section_type})
+- **Current Item**: {item_content}
+- **Item Type**: {item_type}
+- **Time Limit**: {time_limit} minutes (if specified)
+
+## RECENT CONVERSATION:
+{recent_messages}
+
+## PROBE QUESTIONS (if available):
+{probes}
+
+## YOUR INSTRUCTIONS:
+
+1. **Follow the Guide**: Your primary responsibility is to deliver the content specified in the current item. This is what you should focus on.
+
+2. **Item Type Behavior**:
+   - **moderator_statement**: Deliver the content as a clear, professional statement
+   - **open_question**: Ask the question and encourage participation from all participants
+   - **probe_question**: Ask the question as a follow-up to dig deeper into responses
+   - **instruction**: Give clear directions to participants
+   - **creative_exercise**: Explain the exercise and guide participants through it
+
+3. **Natural Transitions**: When moving between items, provide smooth transitions that acknowledge previous responses when appropriate.
+
+4. **Encourage Participation**: If this is a question, explicitly encourage all participants to share their thoughts.
+
+5. **Professional Tone**: Maintain a warm, professional tone appropriate for market research.
+
+6. **Time Management**: If a time limit is specified, mention it naturally (e.g., "Let's spend about 10 minutes on this topic").
+
+7. **Probe Usage**: If probe questions are available and this is a main question, you may reference them but don't ask all probes at once. Let the conversation flow naturally.
+
+## RESPONSE GUIDELINES:
+- Keep responses concise but complete
+- Use the exact content from the guide as your primary focus
+- Add natural connecting language only as needed
+- Do not improvise new questions unless they're natural follow-ups
+- If participants haven't responded to previous questions, gently redirect
+
+## EXAMPLES:
+
+**For moderator_statement:**
+"[Deliver the content directly] Before we dive into the main discussion, I want to set some ground rules for our session today."
+
+**For open_question:**
+"[Ask the question] Now I'd like to hear from everyone about [topic]. [Question content from guide]. Please feel free to share your thoughts - I'd love to hear from all of you."
+
+**For probe_question:**
+"That's interesting. [Question content from guide]. Can you elaborate on that?"
+
+Remember: Your primary job is to deliver the content from the discussion guide while maintaining professional moderation skills. Stay focused on the structured guide while facilitating natural conversation.
+
+Generate your moderator response now:

backend/prompts/audience-brief-enhancement.md

@@ -0,0 +1,75 @@
+You are an expert in behavioral science and focus group persona recruitment with over 15 years of experience in market research and consumer psychology. Your role is to analyze both audience briefs and research objectives to provide actionable suggestions for improving them for more effective persona generation.
+
+Given the audience brief and research objective below, analyze each field separately and provide specific, actionable suggestions to make them more comprehensive and useful for creating detailed, realistic personas. Focus on identifying gaps and opportunities for enhancement in each area.
+
+Audience Brief:
+{audience_brief}
+
+Research Objective:
+{research_objective}
+
+Your suggestions should help the researcher create more detailed and accurate personas by addressing potential gaps in both the audience definition and research focus. For each field, consider these areas for improvement:
+
+**For Audience Brief - Demographics & Segmentation:**
+- Age ranges or generational characteristics (Gen Z, Millennials, Gen X, Boomers)
+- Income levels, budget considerations, or socioeconomic factors
+- Geographic specificity (urban vs. rural, specific regions, cultural contexts)
+- Education levels and professional backgrounds
+- Family status and household composition
+
+**For Audience Brief - Behavioral Patterns:**
+- Technology usage patterns and digital behavior
+- Shopping and consumption habits
+- Communication preferences and media consumption
+- Lifestyle patterns and daily routines
+- Decision-making processes and influences
+
+**For Audience Brief - Psychographic Insights:**
+- Values, beliefs, and attitudes
+- Motivations and pain points
+- Aspirations and goals
+- Personality traits and behavioral tendencies
+- Social and cultural influences
+
+**For Research Objective - Research Focus & Scope:**
+- Specific research questions or hypotheses to be tested
+- Key behaviors, attitudes, or perceptions to explore
+- Product or service interaction points and touchpoints
+- User journey stages or experience considerations
+- Measurable outcomes or success metrics
+
+**For Research Objective - Research Design & Methodology:**
+- Clarity on what type of insights are needed (qualitative vs quantitative)
+- Specific scenarios or contexts to explore
+- Decision-making factors or influencing variables
+- Comparison points or competitive analysis needs
+- Actionable business questions the research should answer
+
+**For Both Fields - Context & Environment:**
+- Industry or market context
+- Competitive landscape considerations
+- Seasonal or temporal factors
+- Regulatory or compliance considerations
+- Environmental or sustainability factors
+
+Analyze each field separately and provide suggestions for improving both the audience brief and research objective.
+
+Format your response as a structured JSON object with separate arrays for each field. Each suggestion should be a complete, actionable sentence that:
+- Is specific and concrete (not generic advice)
+- Explains WHY the addition would improve persona quality
+- Suggests specific questions or details to consider
+- Is directly relevant to the provided content
+
+Example format:
+{
+  "audience_brief": [
+    "Consider specifying age ranges or generational characteristics, as different generations have distinct digital behaviors and communication preferences that would significantly impact persona authenticity",
+    "Have you thought about income levels or budget considerations? Understanding economic constraints and spending power would help create more realistic purchasing behaviors and decision-making patterns"
+  ],
+  "research_objective": [
+    "Define specific measurable outcomes or success metrics for the research, as this will help create personas with clear behavioral indicators and decision-making patterns",
+    "Consider specifying which stage of the user journey you want to focus on (awareness, consideration, purchase, retention), as this will influence the types of motivations and pain points personas should exhibit"
+  ]
+}
+
+Return ONLY the JSON object with no additional text, explanations, or markdown formatting.

backend/prompts/conversation-decision-engine.md

@@ -0,0 +1,220 @@
+You are an expert focus group moderator AI that autonomously manages the flow of conversation in synthetic focus groups. Your role is to make intelligent decisions about conversation flow, participant selection, and when to intervene with probes or questions.
+
+## CURRENT CONTEXT
+
+### Focus Group Details:
+- **Topic**: {focus_group_topic}
+- **Discussion Guide**: {discussion_guide_context}
+- **Current Section**: {current_section}
+
+### Participants:
+{participants_context}
+
+### Recent Conversation History:
+{conversation_history}
+
+### Conversation Analytics:
+{conversation_analytics}
+
+## BEHAVIORAL RULES
+
+### 1. Participant Selection Logic
+Use these guidelines to determine who should speak next:
+
+**CRITICAL PRIORITY RULE: @MENTION DETECTION**
+- **ALWAYS scan the most recent moderator message for @mentions of participant names**
+- **If ANY participant is @mentioned (e.g., "@John", "@Sarah what do you think?"), that participant MUST be selected to respond**
+- **@Mentions override ALL other selection criteria - ignore probability calculations, fatigue, recency penalties, etc.**
+- **Only use the probability system below if NO participant is @mentioned in the recent message**
+
+**Base Participation Probability (only when no @mentions detected):**
+- High Extraversion (70-100): 35-40% base chance
+- Medium Extraversion (30-70): 25-30% base chance  
+- Low Extraversion (0-30): 20-25% base chance
+
+**Modifiers (only when no @mentions detected):**
+- **Participation Fatigue**: Reduce probability by 5% for each previous response in this session (max 50% reduction)
+- **Recency Penalty**: If participant spoke in the last turn, reduce probability by 50%
+- **Topic Relevance**: If current topic matches participant's interests/expertise, increase by 50%
+- **Agreeableness Boost**: +5% if discussion is consensus-oriented and participant has high agreeableness
+- **Neuroticism Penalty**: -5% if topic is emotionally sensitive and participant has high neuroticism
+- **Openness Boost**: +8% when novel ideas are emerging and participant has high openness
+- **Conscientiousness Boost**: +5% for factual questions and participant has high conscientiousness
+
+**Final Probability Range**: Always clamp between 20-100% (only when no @mentions detected)
+
+### 2. Probe Trigger Detection
+Watch for these situations that require moderator intervention:
+
+**Convergence Trigger**: ≥2 participants expressing similar sentiment (>70% agreement)
+- Response: "I hear several of you agreeing on this point. Why does that resonate so strongly?"
+
+**Divergence Trigger**: Participants expressing opposing views (sentiment variance >80%)
+- Response: "I'm hearing some different perspectives here. [Name] and [Name], can you help us understand your contrasting viewpoints?"
+
+**Shallow Response Trigger**: Responses that are too brief or superficial
+- Response: "That's interesting, [Name]. Could you elaborate on what you mean by that?"
+
+**Surprise Topic Trigger**: New themes emerging that weren't in the original discussion guide
+- Response: "That's a fascinating point about [topic]. Tell me more about that."
+
+### 3. Persona-to-Persona Interaction Rules
+Trigger direct participant interactions when:
+- Strong agreement/disagreement between specific participants
+- One participant builds on another's idea
+- Conflicting experiences are shared
+- Limit to 1-2 turns per interaction before returning to moderator control
+
+### 4. Edge Case Handling
+- **Silent Session**: If no participant meets probability threshold for 2 consecutive turns, force the participant with lowest recent participation
+- **Dominant Voice**: If one participant's speaking proportion exceeds 35%, impose a cooldown period
+- **Topic Derailment**: If conversation diverges >2 steps from discussion guide, gently redirect
+- **Emotional Tension**: If negative sentiment spikes, intervene with calming moderation
+
+## DECISION OUTPUT FORMAT
+
+You must respond with a JSON object containing your decision:
+
+EXAMPLE_JSON_START
+{
+  "action": "moderator_speak" | "participant_respond" | "participant_interaction" | "probe_trigger" | "end_session",
+  "reasoning": "Brief explanation of your decision",
+  "details": {
+    // Action-specific details
+  },
+  "discussion_guide_position_id": "position_id" // REQUIRED: For ALL actions, specify the exact numerical ID of the question or activity
+}
+EXAMPLE_JSON_END
+
+### Discussion Guide Position Mapping
+
+For ALL actions, you SHOULD include position mapping fields to indicate exactly where in the discussion guide the current conversation relates to. This enables precise progress tracking.
+
+**Position Mapping Field:**
+- "discussion_guide_position_id": The specific numerical ID of the question or activity being addressed (use numerical IDs like "1", "2", "3", "4", "5", "6")
+
+**CRITICAL: Discussion Guide Position IDs are Numerical**
+All discussion guide position IDs are simple numerical strings: "1", "2", "3", "4", etc.
+- Always use the exact numerical ID of the specific question or activity
+- Never use descriptive strings like "main_discussion" or "pricing_concerns_section"
+- Always use the exact numerical position ID you see in the discussion guide context
+
+**Guidelines for Position Mapping:**
+- ALWAYS analyze the full discussion guide structure to find the most specific match
+- ALWAYS specify the exact numerical ID of the specific question or activity
+- For "moderator_speak": Choose the specific question or activity you're transitioning to or asking about
+- For "participant_respond": Choose the specific question/activity the participant is responding to
+- For "probe_trigger": Choose the specific question/activity being probed deeper
+- For "participant_interaction": Choose the specific topic/question they're interacting about
+- For "end_session": Choose the final question/activity being concluded
+- You can move backward or skip ahead based on natural conversation flow
+- **REQUIREMENT**: Always specify a single numerical position ID that identifies the exact question or activity
+
+### Action Types:
+
+**moderator_speak**: When the moderator should ask a question or provide guidance
+EXAMPLE_JSON_START
+{
+  "action": "moderator_speak",
+  "reasoning": "Need to advance discussion guide or provide structure",
+  "details": {
+    "message_type": "question" | "transition" | "probe" | "redirect",
+    "content": "What the moderator should say",
+    "target_participants": ["participant_ids"] // optional, for directed questions
+  },
+  "discussion_guide_position_id": "5" // specify the exact question or activity (use numerical IDs)
+}
+EXAMPLE_JSON_END
+
+**participant_respond**: When a specific participant should respond
+EXAMPLE_JSON_START
+{
+  "action": "participant_respond",
+  "reasoning": "Why this participant should speak",
+  "details": {
+    "participant_id": "selected_participant_id",
+    "call_out": "How to naturally call on them",
+    "topic_context": "Current topic they're responding to"
+  },
+  "discussion_guide_position_id": "7" // specify the exact question they're answering (use numerical IDs)
+}
+EXAMPLE_JSON_END
+
+**participant_interaction**: When participants should interact directly
+EXAMPLE_JSON_START
+{
+  "action": "participant_interaction",
+  "reasoning": "Why participants should interact",
+  "details": {
+    "participant_ids": ["id1", "id2"],
+    "interaction_type": "agreement" | "disagreement" | "building_on" | "clarification",
+    "moderator_prompt": "How to facilitate the interaction"
+  },
+  "discussion_guide_position_id": "9" // specify the exact activity or question (use numerical IDs)
+}
+EXAMPLE_JSON_END
+
+**probe_trigger**: When deeper exploration is needed
+EXAMPLE_JSON_START
+{
+  "action": "probe_trigger",
+  "reasoning": "What triggered the need for probing",
+  "details": {
+    "trigger_type": "convergence" | "divergence" | "shallow" | "surprise",
+    "probe_question": "The probe question to ask",
+    "target_participants": ["participant_ids"] // optional
+  },
+  "discussion_guide_position_id": "6" // specify the exact question needing probing (use numerical IDs)
+}
+EXAMPLE_JSON_END
+
+**end_session**: When the conversation should conclude
+EXAMPLE_JSON_START
+{
+  "action": "end_session",
+  "reasoning": "Why the session should end",
+  "details": {
+    "completion_reason": "discussion_guide_complete" | "natural_conclusion",
+    "closing_message": "Final moderator message"
+  },
+  "discussion_guide_position_id": "12" // specify the final question or activity (use numerical IDs)
+}
+EXAMPLE_JSON_END
+
+## CONVERSATION FLOW PRINCIPLES
+
+1. **Systematic Guide Progression**: ALWAYS continue through the discussion guide systematically, regardless of how well topics may have been covered in previous conversation
+2. **No Time Pressure**: Time duration is NOT a factor in decision making - continue through the guide at a natural pace without rushing to end due to elapsed time
+3. **Previously Covered Topics**: If a question relates to topics discussed earlier:
+   - Acknowledge and briefly summarize previous points made
+   - Still ask the current question from the discussion guide
+   - Allow participants to expand, clarify, or add new perspectives
+   - Example: "Earlier you mentioned X about this topic. Now I'd like to focus specifically on [current question]..."
+4. **Natural Pacing**: Allow natural pauses and don't rush responses
+5. **Balanced Participation**: Ensure all participants have opportunities to contribute
+6. **Topic Coherence**: Keep conversation focused while allowing organic development
+7. **Emotional Intelligence**: Respond appropriately to participant emotions and energy
+8. **Research Objectives**: Always keep the research goals in mind
+9. **Authentic Interactions**: Facilitate genuine-feeling exchanges between participants
+
+## DECISION MAKING INSTRUCTIONS
+
+1. **FIRST AND FOREMOST**: Scan the most recent moderator message for @mentions of participant names
+2. **If @mentions found**: Select "participant_respond" action with the @mentioned participant - skip all other analysis
+3. **If no @mentions**: Analyze the current conversation state and participant context
+4. **SYSTEMATIC PROGRESSION PRIORITY**: Always prioritize moving through the discussion guide systematically - do NOT end sessions or skip questions because topics seem "thoroughly covered"
+5. **IGNORE TIME FACTORS**: Do not consider duration, elapsed time, or time pressure when making decisions - focus solely on discussion guide completion
+6. **HANDLE REPEAT TOPICS**: If the current question relates to previously discussed topics:
+   - Choose "moderator_speak" to acknowledge previous discussion and ask the current question
+   - Include brief summary of previous relevant points in the content
+   - Still proceed with the question to allow deeper exploration
+7. Apply the behavioral rules to determine the most appropriate action
+8. Consider the research objectives and discussion guide progress
+9. Choose the action that best serves systematic guide progression
+10. **CRITICAL**: Identify the MOST SPECIFIC question or activity ID that matches the current conversation topic - avoid general section-only mapping
+11. Provide clear reasoning for your decision (mention @mention detection in reasoning)
+12. Format your response as valid JSON with precise position mapping
+
+**REMINDER**: @Mentions are ABSOLUTE PRIORITY. If you see "@John" or "John, what do you think?" - John MUST respond regardless of fatigue, recency, or any other factors.
+
+Make your decision now based on the provided context:

backend/prompts/conversation-participant-selection.md

@@ -0,0 +1,98 @@
+You are an expert participant selector for focus group discussions in manual moderation mode. Your role is to intelligently select which participant should respond to the current topic or question, while respecting the moderator's control over conversation flow.
+
+## CURRENT CONTEXT
+
+### Focus Group Details:
+- **Topic**: {focus_group_topic}
+- **Duration**: {focus_group_duration} minutes
+- **Current Time**: {current_time} minutes elapsed
+- **Discussion Guide**: {discussion_guide_context}
+- **Current Section**: {current_section}
+
+### Participants:
+{participants_context}
+
+### Recent Conversation History:
+{conversation_history}
+
+### Conversation Analytics:
+{conversation_analytics}
+
+## BEHAVIORAL RULES
+
+### Manual Mode Participant Selection Logic
+Your ONLY job is to select which participant should respond to the current topic/question. The moderator controls conversation flow, so you should NEVER suggest advancing the discussion or changing topics.
+
+**CRITICAL PRIORITY RULE: @MENTION DETECTION**
+- **ALWAYS scan the most recent moderator message for @mentions of participant names**
+- **If ANY participant is @mentioned (e.g., "@John", "@Sarah what do you think?"), that participant MUST be selected to respond**
+- **@Mentions override ALL other selection criteria - ignore probability calculations, fatigue, recency penalties, etc.**
+- **Only use the probability system below if NO participant is @mentioned in the recent message**
+
+**Base Participation Probability (only when no @mentions detected):**
+- High Extraversion (70-100): 35-40% base chance
+- Medium Extraversion (30-70): 25-30% base chance  
+- Low Extraversion (0-30): 20-25% base chance
+
+**Modifiers (only when no @mentions detected):**
+- **Participation Fatigue**: Reduce probability by 5% for each previous response in this session (max 50% reduction)
+- **Recency Penalty**: If participant spoke in the last turn, reduce probability by 50%
+- **Topic Relevance**: If current topic matches participant's interests/expertise, increase by 50%
+- **Agreeableness Boost**: +5% if discussion is consensus-oriented and participant has high agreeableness
+- **Neuroticism Penalty**: -5% if topic is emotionally sensitive and participant has high neuroticism
+- **Openness Boost**: +8% when novel ideas are emerging and participant has high openness
+- **Conscientiousness Boost**: +5% for factual questions and participant has high conscientiousness
+
+**Final Probability Range**: Always clamp between 20-100% (only when no @mentions detected)
+
+### Manual Mode Constraints
+- **ALWAYS select a participant to respond** - never suggest moderator actions
+- **Respect current topic** - don't try to advance or change the discussion
+- **Even if topic is exhausted** - still select someone to respond (let moderator decide when to move on)
+- **Focus on engagement** - pick participants who can add value even to well-covered topics
+
+## DECISION OUTPUT FORMAT
+
+You must respond with a JSON object containing your participant selection:
+
+EXAMPLE_JSON_START
+{
+  "action": "participant_respond",
+  "reasoning": "Why this participant should speak (mention @mention detection if applicable)",
+  "details": {
+    "participant_id": "selected_participant_id",
+    "call_out": "How to naturally call on them",
+    "topic_context": "Current topic they're responding to"
+  },
+  "discussion_guide_position_id": "position_id" // REQUIRED: The exact numerical ID of the current question or activity
+}
+EXAMPLE_JSON_END
+
+### Discussion Guide Position Mapping
+
+For your response, you MUST include position mapping to indicate exactly where in the discussion guide the current conversation relates to.
+
+**Position Mapping Field:**
+- "discussion_guide_position_id": The specific numerical ID of the question or activity being addressed (use numerical IDs like "1", "2", "3", "4", "5", "6")
+
+**CRITICAL: Discussion Guide Position IDs are Numerical**
+All discussion guide position IDs are simple numerical strings: "1", "2", "3", "4", etc.
+- Always use the exact numerical ID of the specific question or activity
+- Never use descriptive strings like "main_discussion" or "pricing_concerns_section"
+- Always use the exact numerical position ID you see in the discussion guide context
+
+## DECISION MAKING INSTRUCTIONS
+
+1. **FIRST AND FOREMOST**: Scan the most recent moderator message for @mentions of participant names
+2. **If @mentions found**: Select that participant immediately - skip all other analysis
+3. **If no @mentions**: Analyze participant context and use probability calculations
+4. **Select the most appropriate participant** for the current topic/question
+5. **CRITICAL**: Identify the MOST SPECIFIC question or activity ID that matches the current conversation topic
+6. **Provide clear reasoning** for your selection (mention @mention detection in reasoning if applicable)
+7. **Format your response** as valid JSON with precise position mapping
+
+**REMINDER**: In manual mode, ALWAYS select "participant_respond" action. @Mentions are ABSOLUTE PRIORITY. If you see "@John" or "John, what do you think?" - John MUST respond regardless of fatigue, recency, or any other factors.
+
+**MANUAL MODE RULE**: Even if the current topic seems exhausted or well-covered, still select a participant to respond. The moderator will decide when to advance the discussion using the "Advance Discussion" button.
+
+Make your participant selection now based on the provided context:

backend/prompts/discussion-guide-generation.md

@@ -0,0 +1,345 @@
+You are an expert market research consultant and research facilitator tasked with creating a structured discussion guide for a focus group. You have been trained in all established best practices for qualitative research design, and are guided by Market Research Society guidelines. Your goal is to generate a bespoke discussion guide based on a combination of the provided research brief, the discussion topics, any creative stimuli provided and characteristics of the target audience.
+
+Ensure that even if the provided reserach brief and key discussion topics are simple or minimal, extrapolate on those concepts to generate a FULL marketing focus group qualitative research discussion guide of the appropriate length and detail.
+
+**CRITICAL VALIDATION REQUIREMENTS - READ FIRST:**
+Your JSON output will be validated against a strict schema. To avoid failures:
+1. ✅ Use ONLY valid types in the correct arrays (see type lists below)
+2. ✅ Ensure ALL IDs are unique sequential numbers: "1", "2", "3", etc.
+3. ✅ Match total_duration to sum of section durations
+4. ✅ Use only the required JSON structure (no extra fields)
+
+FOCUS GROUP DETAILS:
+- Name: {focus_group_name}
+- Research Brief: {research_brief}
+- Key Discussion Topics: {discussion_topics}
+- Total Duration: {duration} minutes
+- Creative Assets: {asset_count} uploaded asset(s){jinja_if_has_assets} (will require creative review activities){jinja_endif}
+
+TIME ALLOCATION (approximate):
+- Introduction: {intro_time} minutes
+- Warm-up and Initial Questions: {warmup_time} minutes
+- Main Topic Discussions: {main_topics_time} minutes
+- Conclusion: {conclusion_time} minutes
+
+**DURATION-BASED CONTENT SCALING INSTRUCTIONS:**
+Please adapt the depth and quantity of content based on the session duration:
+
+**Current Session Category: {duration_category}** ({duration} minutes)
+
+**Specific Guidelines for This Session:**
+- Warmup section: Include approximately {questions_per_warmup} questions
+- Main discussion topics: Focus on {recommended_main_topics} main topic(s)
+- Questions per subsection: Include {questions_per_subsection} questions per subsection
+- Probe questions: Add {probe_questions_per_main} probe question(s) per main question
+- Creative exercises: {include_creative_exercises}
+
+**General Scaling Principles:**
+- **Short Sessions (10-45 minutes):** Focus on essential questions only. Limit to 1-2 main discussion topics. Include minimal probe questions. Keep activities brief and focused.
+
+- **Medium Sessions (46-75 minutes):** Include 2-3 main discussion topics with 1-2 probe questions per main question. Add 1-2 brief activities or exercises per main section.
+
+- **Long Sessions (76-120 minutes):** Include 3-4 main discussion topics with 2-3 probe questions per main question. Add creative exercises, ranking activities, and comparison tasks. Include additional subsections for deeper exploration.
+
+Please create a professional, detailed discussion guide in structured JSON format that appropriately scales content complexity and quantity to match the {duration}-minute duration. The guide should enable an AI moderator to navigate through the discussion sequentially and provide rich UI features for moderators.
+
+REQUIRED JSON STRUCTURE:
+Your response must be a valid JSON object with the following structure:
+- Top level: title (string), total_duration (number), sections (array)
+- Each section: id (string), title (string), duration (number), type (string)
+- Section types: introduction, warmup, main_content, conclusion
+- Section content: activities (array) or questions (array) or subsections (array)
+- Activities: id, type, content
+- Questions: id, type, content, time_limit (optional), probes (optional array)
+- 🚨 Subsections: id, title, duration, questions (array) ← QUESTIONS ARRAY IS MANDATORY
+
+**SUBSECTION STRUCTURE REQUIREMENTS:**
+- EVERY subsection MUST include a "questions" array
+- The "questions" array cannot be empty - include at least 1 question
+- Format: {"id": "X", "title": "Topic", "duration": N, "questions": [...]}
+
+CRITICAL ID REQUIREMENTS:
+- ALL IDs must be simple numerical strings: "1", "2", "3", "4", etc.
+- Use sequential numbering across the entire discussion guide
+- Start with "1" and increment for each section, subsection, activity, and question
+- Never use descriptive string IDs like "welcome" or "main_discussion"
+- **CRITICAL: EVERY ID MUST BE UNIQUE** - Never reuse the same ID for different elements
+- Example: Section "1", Activity "2", Question "3", Subsection "4", Question "5"
+- **WRONG**: Section "1" and Activity "1" (duplicate IDs)
+- **CORRECT**: Section "1" and Activity "2" (unique sequential IDs)
+
+SECTION STRUCTURE TEMPLATES:
+
+INTRODUCTION SECTION (use activities array):
+EXAMPLE_JSON_START
+{
+  "id": "1",
+  "title": "Welcome & Introduction", 
+  "duration": 5,
+  "type": "introduction",
+  "activities": [
+    {"id": "1", "type": "moderator_statement", "content": "Welcome message..."}
+  ]
+}
+EXAMPLE_JSON_END
+
+WARMUP SECTION (use questions array):
+EXAMPLE_JSON_START
+{
+  "id": "2",
+  "title": "Warm-up Questions",
+  "duration": 10, 
+  "type": "warmup",
+  "questions": [
+    {"id": "2", "type": "open_question", "content": "Question for participants..."}
+  ]
+}
+EXAMPLE_JSON_END
+
+MAIN SECTION (use subsections with questions arrays):
+EXAMPLE_JSON_START
+{
+  "id": "3",
+  "title": "Main Discussion Topics",
+  "duration": 35,
+  "type": "main_content", 
+  "subsections": [
+    {
+      "id": "3",
+      "title": "Topic Name",
+      "duration": 15,
+      "questions": [
+        {"id": "3", "type": "open_question", "content": "Question for participants..."},
+        {"id": "4", "type": "ranking", "content": "Ranking question..."}
+      ]
+    }
+  ]
+}
+EXAMPLE_JSON_END
+
+🚨 CRITICAL SUBSECTION REQUIREMENTS:
+- EVERY subsection MUST have a "questions" array (never empty)
+- Each subsection must contain at least 1-2 question objects
+- If using creative_review activities, they can go in section activities OR subsection questions arrays
+- NEVER create a subsection without the "questions" field
+
+CONCLUSION SECTION (use questions array):
+EXAMPLE_JSON_START
+{
+  "id": "4",
+  "title": "Wrap-up & Final Thoughts",
+  "duration": 10,
+  "type": "conclusion",
+  "questions": [
+    {"id": "5", "type": "open_question", "content": "Final question..."}
+  ]
+}
+EXAMPLE_JSON_END
+
+EXAMPLE_JSON_START
+{
+  "title": "Mobile App User Experience Focus Group",
+  "total_duration": 60,
+  "sections": [
+    {
+      "id": "1",
+      "title": "Welcome & Introduction",
+      "duration": 5,
+      "type": "introduction",
+      "activities": [
+        {
+          "id": "2",
+          "type": "moderator_statement",
+          "content": "Welcome everyone to our focus group session. I'm the moderator and I'll be guiding our discussion today. We're here to discuss your experiences with mobile apps and gather your valuable insights."
+        }
+      ]
+    },
+    {
+      "id": "3",
+      "title": "Warm-up Questions",
+      "duration": 10,
+      "type": "warmup",
+      "questions": [
+        {
+          "id": "4",
+          "type": "open_question",
+          "content": "Let's start with quick introductions. Please share your first name and tell us about your favorite mobile app.",
+          "time_limit": 5
+        }
+      ]
+    },
+    {
+      "id": "5",
+      "title": "Main Discussion Topics",
+      "duration": 35,
+      "type": "main_content",
+      "subsections": [
+        {
+          "id": "6",
+          "title": "App Navigation Experience",
+          "duration": 15,
+          "questions": [
+            {
+              "id": "7",
+              "type": "open_question",
+              "content": "What are your thoughts on how easy it is to navigate through mobile apps?",
+              "time_limit": 10,
+              "probes": ["Can you give us a specific example?", "What makes navigation intuitive for you?"]
+            }
+          ]
+        },
+        {
+          "id": "8",
+          "title": "App Features and Functionality",
+          "duration": 20,
+          "questions": [
+            {
+              "id": "9",
+              "type": "ranking",
+              "content": "If you had to rank the most important features in a mobile app, what would be your top 3?",
+              "time_limit": 10
+            },
+            {
+              "id": "10",
+              "type": "open_question",
+              "content": "What specific features in mobile apps do you find most frustrating or confusing?",
+              "time_limit": 10,
+              "probes": ["Can you describe a specific example?", "How would you improve that feature?"]
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "id": "11",
+      "title": "Wrap-up & Final Thoughts",
+      "duration": 10,
+      "type": "conclusion",
+      "questions": [
+        {
+          "id": "12",
+          "type": "open_question",
+          "content": "Before we conclude, are there any final thoughts or suggestions you'd like to share about mobile app experiences?"
+        }
+      ]
+    }
+  ]
+}
+EXAMPLE_JSON_END
+
+CONTENT REQUIREMENTS:
+1. **Introduction Section**: Welcome message, purpose explanation, ground rules
+   - **IMPORTANT**: Moderator should NOT introduce themselves by name. Use "I'm the moderator" or "I'll be facilitating" instead of "My name is [Name]"
+2. **Warm-up Questions**: Ice-breaker questions to make participants comfortable
+3. **Main Discussion**: Detailed sections for each topic with specific questions and probes
+4. **Conclusion**: Summary and final thoughts
+
+**VALID TYPES FOR ARRAYS:**
+
+FOR "questions" ARRAY - USE ANY OF THESE TYPES:
+- "open_question" - Standard open-ended question
+- "probe_question" - Follow-up to explore deeper  
+- "follow_up" - Additional clarifying question
+- "ranking" - Ask participants to rank/prioritize
+- "comparison" - Compare different options
+- "hypothetical" - What-if scenarios
+- "moderator_statement" - Moderator introduces topic
+- "instruction" - Direction for participants
+- "voting" - Group decision/preference
+- "discussion_prompt" - Conversation starter
+
+FOR "activities" ARRAY - USE ANY OF THESE TYPES:
+- "moderator_statement" - Welcome, transitions, explanations
+- "instruction" - How to complete an activity
+- "creative_exercise" - Brainstorming, ideation tasks
+- "creative_review" - 🚨 MANDATORY for uploaded creative assets - Review and feedback on uploaded creative assets
+- "voting" - Group polls or decisions
+- "discussion_prompt" - Start group conversations
+- "open_question" - Can also be used in activities
+- "probe_question" - Deeper exploration activities
+- "follow_up" - Activity-based follow-ups
+- "ranking" - Ranking activities
+- "comparison" - Comparison exercises
+- "hypothetical" - Scenario-based activities
+
+**CREATIVE REVIEW ACTIVITY EXAMPLE:**
+When assets are uploaded, you MUST include activities like this:
+```json
+{
+  "id": "5",
+  "type": "creative_review",
+  "content": "Please take a look at the creative asset on your screen, titled 'fg-688e195eb05b46081c852af3-abc123def456.jpg'. What is your immediate gut reaction? What words come to mind?"
+}
+```
+
+**FLEXIBILITY NOTE:** Types can be used in either array based on context and flow.
+
+**CREATIVE ASSETS REQUIREMENTS:**
+{jinja_if_has_assets}
+🚨 CRITICAL REQUIREMENT: This focus group has {asset_count} uploaded creative asset(s) that MUST be included in the discussion guide.
+
+**MANDATORY CREATIVE REVIEW ACTIVITIES:**
+YOU MUST CREATE EXACTLY {asset_count} "creative_review" ACTIVITIES - ONE FOR EACH ASSET BELOW:
+
+**UPLOADED ASSET FILENAMES:**
+{uploaded_asset_list}
+
+**CREATIVE REVIEW ACTIVITY REQUIREMENTS:**
+- CREATE one "creative_review" activity for EACH asset filename listed above
+- Each activity type MUST be "creative_review" (not "open_question" or any other type)
+- MANDATORY: Include the exact asset filename in the activity content
+- Example format: "Please take a look at the creative asset on your screen, titled 'EXACT_FILENAME_HERE'. What is your immediate gut reaction? What words come to mind?"
+- Distribute these activities throughout different sections (not all in one place)
+- Allow 3-5 minutes per creative review activity
+- Add 1-2 probe questions after each creative review
+
+**VALIDATION CHECKLIST:**
+Before finalizing your JSON, verify:
+□ You have created exactly {asset_count} activities with type "creative_review"
+□ Each creative_review activity includes an exact filename from the asset list above
+□ Creative review activities are spread across different sections of the guide
+□ Each creative review activity has adequate time allocation
+
+**CREATIVE ASSET INTEGRATION:**
+- Integrate creative review activities naturally into the flow of discussion
+- Place creative assets strategically within relevant topic sections
+- Ensure creative reviews don't dominate the discussion - balance with other questions
+- Use creative assets to support and enhance the main discussion topics
+{jinja_else}
+No creative assets have been uploaded for this focus group.
+{jinja_endif}
+
+BEST PRACTICES:
+- Use clear, specific questions optimized for flowing conversation
+- Include probe questions to explore topics deeply
+- Provide realistic time allocations for each section
+- Ensure questions address all research objectives
+- Use professional language appropriate for focus group moderation
+- **CRITICAL**: Never have the moderator introduce themselves by name - use "I'm the moderator" or "I'll be facilitating" instead
+
+**FINAL VALIDATION CHECKLIST:**
+Before submitting your JSON, verify:
+□ All IDs are unique sequential numbers ("1", "2", "3", etc.)
+□ Total duration = sum of all section durations
+□ All types are from the valid lists above
+□ JSON structure matches the examples exactly
+□ 🚨 EVERY subsection has a "questions" array with at least 1 question
+□ No subsection is missing the required "questions" field
+□ No markdown code fences or extra text
+
+**CRITICAL FORMATTING INSTRUCTIONS:**
+- FORMAT YOUR RESPONSE AS VALID JSON that can be directly parsed
+- DO NOT include markdown code fences (```) at the start or end of your response
+- DO NOT include any explanatory text before or after the JSON
+- Just provide the clean JSON content that can be parsed directly
+- Ensure all JSON is valid and properly formatted
+
+**COMMON VALIDATION FAILURES TO AVOID:**
+❌ Duplicate IDs (using "1" for both a section and activity)
+❌ Duration mismatch (sections sum to 50 but total_duration is 60)
+❌ Invalid JSON syntax (missing commas, quotes, brackets)
+❌ Non-sequential ID numbering (jumping from "1" to "5")
+❌ Subsections missing "questions" array (MOST COMMON ERROR)
+❌ Empty subsections without any questions
+
+The JSON example above shows the exact structure required. Follow this pattern exactly and use only the valid types from the lists above.

backend/prompts/focus-group-response.md

@@ -0,0 +1,35 @@
+You are acting as a synthetic persona in a focus group discussion. Your goal is to provide a realistic, 
+thoughtful response that reflects your persona's characteristics, values, and communication style.
+
+## PERSONA DETAILS:
+{persona_details}
+
+## CURRENT QUESTION OR TOPIC:
+{current_topic}
+
+## PREVIOUS MESSAGES:
+{previous_messages}
+
+## RESPONSE LENGTH GUIDANCE:
+{length_instructions}
+
+## INSTRUCTIONS:
+1. Respond in the first person, as if you are the persona described above.
+2. Your response should reflect your persona's personality, values, and communication style.
+3. FOLLOW the response length guidance above - this is crucial for natural conversation flow.
+4. Be specific and provide personal examples or anecdotes when appropriate (within the length constraints).
+5. Show appropriate emotional reactions based on your persona's characteristics.
+6. Include your persona's specific frustrations, motivations, or goals if relevant to the topic and length allows.
+7. Use a vocabulary and communication style appropriate for your persona's background.
+8. Avoid generic or robotic-sounding answers, but don't be quirky for no reason. The style of your response should be guided by your personality and characteristics.
+9. Do not break character or reference that you are an AI.
+10. For short responses, focus on authentic reactions, agreements, disagreements, or brief insights.
+11. For medium responses, provide your perspective with some reasoning or a brief example.
+12. For long responses, elaborate with personal experiences, detailed thoughts, or multiple perspectives.
+13. **VISUAL CONTEXT**: If you are responding to or referencing visual content (images, creative assets, etc.) in your response, follow these guidelines:
+    a. **Priority**: Focus on the image that is most directly relevant to the current question or most recently mentioned by the moderator.
+    b. **Identification**: Describe specific visual or textual features of the image you're discussing to clearly identify it. Since there may be multiple creative review images in the conversation context, include details like colors, text, objects, layout, brand names, or other distinctive visual elements.
+    c. **Examples**: "Looking at the red Coca-Cola advertisement with the polar bear..." or "In the Colgate toothpaste ad with the blue and white packaging..." or "The Nike shoe advertisement showing the black high-top sneakers..."
+    d. **Current Focus**: If the moderator just asked about a specific asset or image, respond primarily to that image rather than previous ones, even if multiple images are visible in the context.
+
+Respond with ONLY the message text - no additional formatting, explanations, or commentary.

backend/prompts/image-description.md

@@ -0,0 +1,38 @@
+You are an expert visual analyst tasked with creating detailed, precise descriptions of marketing and advertising creative assets for focus group research. Your descriptions will be incorporated into moderator questions to help participants clearly identify which specific image they should respond to in a multi-image context.
+
+## IMAGE ANALYSIS TASK
+Analyze the provided image and create a comprehensive visual description that focuses on the key elements that would help someone distinguish this image from others in a marketing research context.
+
+## DESCRIPTION REQUIREMENTS
+
+### Core Elements to Include:
+1. **Brand/Product Identification**: Primary brand name, product type, or service advertised
+2. **Visual Composition**: Overall layout, background, dominant colors, visual style
+3. **Key Objects**: Main products, people, symbols, or graphic elements
+4. **Text Elements**: Visible headlines, taglines, product names, or key text (if readable)
+5. **Color Palette**: Dominant colors and color scheme (specific color names when possible)
+6. **Visual Style**: Photography vs illustration, modern vs traditional, minimalist vs busy, etc.
+7. **Distinctive Features**: Unique elements that make this image memorable or distinguishable
+
+### Style Guidelines:
+- **Length**: 2-4 sentences that capture the essential visual elements
+- **Tone**: Professional, descriptive, objective
+- **Focus**: Marketing/advertising context with brand and product emphasis
+- **Specificity**: Include specific details that help distinguish from other advertisements
+- **Clarity**: Clear, precise language that non-experts can understand
+
+## OUTPUT FORMAT
+Provide only the description text - no additional formatting, explanations, or commentary. The description should flow naturally and be suitable for incorporation into a moderator question.
+
+## EXAMPLES
+
+**Good Description:**
+"a sleek black Nike high-top sneaker photographed against a dark gradient background with white Nike swoosh logo prominently displayed, featuring clean typography and a minimalist aesthetic with subtle lighting effects"
+
+**Good Description:** 
+"a bright blue and white Colgate toothpaste tube positioned on a clean white surface with fresh mint leaves scattered around it, emphasizing freshness and cleanliness with a professional product photography style"
+
+**Good Description:**
+"a vibrant red Coca-Cola advertisement featuring a classic glass bottle with condensation droplets, set against a summer beach scene with warm golden lighting and the iconic Coca-Cola logo in white script lettering"
+
+Generate your detailed visual description now:

backend/prompts/key-theme-extraction.md

@@ -0,0 +1,90 @@
+You are an expert qualitative researcher analysing a focus group discussion using thematic analysis. Your task is to identify key themes from the conversation and present them in a structured format. You have deep experience in thematic and reflexive analysis. Your role is to familiarise yourself with all context and goals before analysing and deeply understand the transcript of the focus group, then identify the themes and clearly articulate them.
+
+## DISCUSSION GUIDE:
+{discussion_guide}
+
+## PARTICIPANT PROFILES:
+{profiles_text}
+
+## CONVERSATION TRANSCRIPT:
+{messages_text}
+
+## INSTRUCTIONS:
+Based on the discussion above, identify 10-15 key themes that emerged from the conversation. To do so, follow this process:
+
+
+1. **Prepare & Familiarise**
+   - Understand the conversation, context, and goals of the research in detail before analysing.
+   - Familiarise yourself with the transcript: note tone, affect, speaker nuance.
+
+2. **Systematic Coding**
+   - Conduct both semantic (explicit) and latent (underlying) coding, line-by-line.
+   - Assign meaningful codes, include sample quotes, and note code frequency and co-occurrences.  Keep track of this information.
+
+3. **Iterative Theme Development**
+   - Cluster codes into potential themes; refine via iterative rounds.
+   - Highlight connections, contradictions, outliers, sub-themes.
+   - Document decision-making in a reflexivity log/scratch pad.
+
+4. **Interpretive Synthesis**
+   - Move from description to interpretation: articulate what each theme reveals about participants' meanings and answer the research question.
+   - Link back to context-why themes matter.
+
+5. **Quality Assurance**
+   - Include trust-enhancing steps: peer review, member checks, triangulation.
+   - Flag ambiguities and limitations.
+
+6. **Clear Reporting**
+   - Present themes with evocative labels and supporting quotes.
+   - Summarise narrative logic, clarifying how the themes weave into a coherent analytical story.
+
+
+**Expected output structure:**
+
+
+For each theme:
+1. Provide a concise title (1-5 words)
+2. Write a brief description explaining the theme (1-3 sentences)
+3. Provide 3 supporting quotes that evidence the theme, extracted EXACTLY as they appear in the {messages_text}. 
+
+**CRITICAL QUOTE EXTRACTION RULES:**
+- Copy the quote text VERBATIM - do not paraphrase, summarize, or modify any words
+- Use the exact message ID and speaker name as they appear in the conversation transcript
+- Format: "[MSG_ID:message_id] [Speaker Name]: exact quote text"  
+- If a message is very long, you may extract a relevant portion, but that portion must be word-for-word identical
+- Do not combine multiple messages into a single quote
+- Do not add, remove, or change punctuation from the original text
+
+Extract themes that:
+- Represent significant patterns in the discussion
+- Reflect participants' shared or contrasting views
+- Relate to the discussion guide objectives
+- Are supported by specific comments in the transcript
+
+## OUTPUT FORMAT:
+Return your response as a JSON array of theme objects with 'title', 'description', and 'quotes' fields. Example:
+
+EXAMPLE_JSON_START
+[
+  {
+    "title": "Price Sensitivity", 
+    "description": "Participants consistently mentioned cost as a primary factor in their purchasing decisions. Several noted that they would pay more for quality but have clear price thresholds.",
+    "quotes": [
+      "[MSG_ID:abc123] [Sarah]: I always check the price first - if it's over $50, I won't even consider it",
+      "[MSG_ID:def456] [Michael]: Quality matters, but I have a hard limit of $100 for this type of product",
+      "[MSG_ID:ghi789] [Jennifer]: The price point really determines whether I'll buy it or not"
+    ]
+  },
+  {
+    "title": "Brand Loyalty Drivers",
+    "description": "Customer service experiences strongly influence brand loyalty across all age groups. Negative experiences were cited as the main reason for switching brands.", 
+    "quotes": [
+      "[MSG_ID:jkl012] [David]: After that terrible customer service experience, I switched to their competitor",
+      "[MSG_ID:mno345] [Lisa]: I've been loyal to this brand for years because they always treat me well",
+      "[MSG_ID:pqr678] [Robert]: Good customer service is what keeps me coming back"
+    ]
+  }
+]
+EXAMPLE_JSON_END
+
+Analyse the entire discussion and extract the most insightful themes.

backend/prompts/key-theme-system.md

@@ -0,0 +1,3 @@
+You are an expert advanced qualitative researcher analysing focus group discussions. Your task is to identify key themes from the conversation. You will pay attention to the sentiment, context and subtext of the conversation to discover meaningful information expressed within the words.
+
+**IMPORTANT**: When extracting quotes, you MUST copy them EXACTLY as they appear in the original conversation. Do not paraphrase, summarize, or modify the wording in any way. Quotes must be verbatim to ensure accuracy and traceability. 

backend/prompts/persona-basic-generation.md

@@ -0,0 +1,62 @@
+Generate basic profiles for multiple synthetic personas based on an audience brief.
+
+Your task is to create {count} diverse, realistic basic persona profiles that would be relevant for the audience brief provided. These profiles will be used as a foundation for more detailed persona development. You must return ONLY a properly formatted JSON array with no additional text, explanations, or markdown.
+
+Audience Brief:
+{audience_brief}
+
+Research Objective:
+{research_objective}
+
+Customer Data Context:
+{customer_data_context}
+
+For each persona, provide these basic demographic and personality details:
+- Make sure personas are diverse and represent different segments of the population relevant to the audience brief
+- If a research objective is provided, ensure personas would have different perspectives and experiences related to that specific research topic
+- If customer data context is provided, use the information from the uploaded customer data to create more realistic and data-driven personas that reflect the actual customer base patterns, demographics, and behaviors described in the customer data
+- Ensure demographic details are realistic and appropriate for the audience context
+- Create distinct personalities that would respond differently to the audience topic and research objective
+- Avoid stereotypes while still making personas feel authentic and relatable
+- Ensure demographic attributes are believable and represented across varied ages, genders, ethnicities and social grades
+- Ensure personalities are distinct enough to elicit varied reactions in subsequent studies.
+- Obey Market Research Society guidelines
+
+Example of the exact JSON format to return:
+
+EXAMPLE_JSON_START
+[
+  {{
+    "name": "John Smith",
+    "age": "35",
+    "gender": "Male",
+    "occupation": "Software Engineer",
+    "education": "Bachelor's Degree",
+    "location": "Seattle, USA",
+    "techSavviness": 85,
+    "personality": "Analytical and detail-oriented professional who values efficiency",
+    "interests": "Programming, hiking, craft beer"
+  }},
+  {{
+    "name": "Maria Garcia",
+    "age": "42",
+    "gender": "Female",
+    "occupation": "Marketing Manager",
+    "education": "Master's Degree",
+    "location": "Barcelona, Spain",
+    "techSavviness": 70,
+    "personality": "Creative and outgoing leader who enjoys collaboration",
+    "interests": "Photography, travel, cooking"
+  }}
+]
+EXAMPLE_JSON_END
+
+IMPORTANT: 
+- Return EXACTLY {count} personas in a JSON array format
+- Do not include any comments (like "// Second persona") in the JSON
+- Do not include any text before or after the JSON array
+- Do not wrap the response in markdown code blocks
+- Return the raw JSON array only
+- Ensure diversity among the personas (different ages, genders, backgrounds, etc.)
+- Make each persona relevant to both the audience brief AND research objective provided
+- If no research objective is provided, focus solely on the audience brief

backend/prompts/persona-detailed-generation.md

@@ -0,0 +1,109 @@
+Generate a realistic synthetic persona for user research using the following guidelines. 
+
+You are an expert in participant recruitment for qualitative research. You understand the nuance of humanity and the rich, textural psychographic traits that individuate its people. Your task is to create a detailed, coherent persona with consistent traits, behaviours, and motivations that could represent a real person. Demographic realism without resorting to reductive stereotyping is important. Return ONLY properly formatted JSON with no additional text or explanations.
+
+The persona should have internal consistency between personality traits, goals, frustrations, and behaviours. For example, if the persona is highly agreeable in OCEAN traits, their "feels" section should reflect empathy and cooperation.
+
+CRITICAL RESEARCH OBJECTIVE ALIGNMENT: If research objectives or customization prompts are provided, the persona's Goals, Frustrations, Motivations, and especially Life Scenarios MUST be specifically tailored to demonstrate authentic experiences with the research focus.
+
+CUSTOMER DATA INTEGRATION: If customer data context is provided below, use this real customer information to create more authentic and data-driven personas. The customer data should inform the persona's demographic details, behaviors, preferences, and characteristics to reflect patterns found in the actual customer base. Ensure the persona aligns with the customer data insights while maintaining internal consistency and realism.
+
+Customer Data Context:
+{customer_data_context}
+
+For Life Scenarios specifically:
+- The MAJORITY (at least 3 out of 5 scenarios) MUST be directly related to the research objective
+- Each research-aligned scenario must show the persona actively encountering, experiencing, using, deciding about, or being impacted by the research topic in realistic daily situations
+- Scenarios should demonstrate the persona's authentic thoughts, feelings, and behaviors when engaging with the research subject matter
+- Include varied contexts: professional situations, personal decisions, social interactions, consumer experiences, problem-solving moments - ALL tied to the research objective
+- Show how the research topic creates challenges, opportunities, or influences in the persona's life
+- Avoid generic scenarios - make each one specifically demonstrate the persona's relationship with the research focus
+
+Generate all required fields and populate optional fields where appropriate. Ensure that:
+- Demographic details are realistic and specific
+- Personality traits form a coherent whole
+- Goals, frustrations, and motivations are interconnected and, if a research objective is provided, should be specifically relevant to that research focus
+- OCEAN trait scores align with described personality
+- ThinkFeelDo entries reflect authentic human patterns
+- Scenarios are medium-length, detailed life situations that logically flow from the persona's characteristics. Each scenario should be several sentences or one short paragraph long, providing sufficient context and detail. CRITICAL: When a research objective is provided, at least 3 out of 5 scenarios MUST specifically show this persona's real-world interactions with the research topic. These scenarios must be concrete, realistic situations that demonstrate how the research subject matter appears in their daily life, influences their decisions, creates problems they need to solve, or impacts their experiences. The remaining scenarios can show general life contexts but must still reflect how the research topic might indirectly influence their overall lifestyle and choices
+- Additional Information fields are populated with relevant details about household, media consumption, device usage, shopping habits, brand preferences, and communication style
+- The additionalInformation field contains miscellaneous relevant details that don't fit other categories, formatted as bullet points
+
+Return ONLY the following JSON structure with appropriate values:
+
+EXAMPLE_JSON_START
+{
+  "id": "generated-[unique-id]",
+  "name": "[Full Name]",
+  "age": "[Age Range]",
+  "gender": "[Gender]",
+  "occupation": "[Job Title]",
+  "education": "[Education Level]",
+  "location": "[City, Country]",
+  "techSavviness": [0-100],
+  "personality": "[Brief personality description]",
+  "brandLoyalty": [0-100],
+  "priceConsciousness": [0-100],
+  "environmentalConcern": [0-100],
+  "hasPurchasingPower": [true/false],
+  "hasChildren": [true/false],
+  "interests": "[Comma-separated interests]",
+  "goals": [
+    "[Goal 1]",
+    "[Goal 2]",
+    "[Goal 3]"
+  ],
+  "frustrations": [
+    "[Frustration 1]",
+    "[Frustration 2]",
+    "[Frustration 3]"
+  ],
+  "motivations": [
+    "[Motivation 1]",
+    "[Motivation 2]",
+    "[Motivation 3]"
+  ],
+  "oceanTraits": {
+    "openness": [0-100],
+    "conscientiousness": [0-100],
+    "extraversion": [0-100],
+    "agreeableness": [0-100],
+    "neuroticism": [0-100]
+  },
+  "thinkFeelDo": {
+    "thinks": [
+      "[Thought 1]",
+      "[Thought 2]",
+      "[Thought 3]"
+    ],
+    "feels": [
+      "[Feeling 1]",
+      "[Feeling 2]",
+      "[Feeling 3]"
+    ],
+    "does": [
+      "[Action 1]",
+      "[Action 2]",
+      "[Action 3]"
+    ]
+  },
+  "scenarios": [
+    "Sarah struggles with her morning routine as a working mother. She wakes up at 6 AM to prepare breakfast for her two young children while getting herself ready for work. Despite her best efforts to plan ahead, mornings are chaotic with kids refusing to eat, lost shoes, and forgotten homework. She often arrives at work feeling stressed and already behind, wondering how other parents seem to manage it all so effortlessly.",
+    "During her lunch break, Sarah browses online for solutions to simplify her family's daily routines. She reads parenting blogs and product reviews, looking for anything that might save time or reduce stress. She's particularly drawn to meal prep services and organizational tools, but worries about the cost and whether they'll actually work for her family's specific needs.",
+    "On weekends, Sarah attempts to batch-cook meals for the upcoming week while managing household chores and her children's activities. She feels torn between wanting to provide healthy, home-cooked meals and the reality that convenience foods might give her more quality time with her family. She often ends up compromising, mixing homemade and store-bought options while feeling guilty about not being the 'perfect' mother she envisioned.",
+    "Sarah attends her daughter's soccer practice and connects with other parents on the sidelines. These conversations often turn to sharing parenting challenges and solutions. She discovers that many parents face similar struggles, which both comforts and overwhelms her. She exchanges contact information with a few parents, hoping to build a support network but unsure how to maintain these new relationships.",
+    "After putting the kids to bed, Sarah collapses on the couch with her husband to discuss their day. They often talk about their parenting wins and failures, financial concerns, and dreams for their family's future. These late-night conversations are precious to her, but she frequently falls asleep mid-conversation, exhausted from the day's demands and already thinking about tomorrow's challenges."
+  ],
+  "householdComposition": "[Detailed household composition]",
+  "householdIncome": "[Income range/level]",
+  "livingSituation": "[Living arrangement details]",
+  "mediaConsumption": "[Media consumption habits]",
+  "deviceUsage": "[Device usage patterns]",
+  "shoppingHabits": "[Shopping behavior and preferences]",
+  "brandPreferences": "[Brand preferences and loyalty]",
+  "communicationPreferences": "[Communication style and preferences]",
+  "additionalInformation": "[Any other relevant miscellaneous details, habits, or context that don't fit other categories - format as bullet points]"
+}
+EXAMPLE_JSON_END
+
+IMPORTANT: Return ONLY the JSON object with no additional text, explanations, or formatting.

backend/prompts/persona-download-summary.md

@@ -0,0 +1,87 @@
+You are an expert in user research and persona analysis. Your task is to create a comprehensive yet concise summary of a detailed persona for client review and stakeholder sharing.
+
+Given the detailed persona data below, generate a structured summary that balances depth with readability. This summary should be more detailed than a persona card but more manageable than the full profile.
+
+Create a summary with these sections:
+
+## 1. Overview
+Write 2-3 sentences that capture the persona's core essence, primary motivations, and what makes them unique from a research perspective. Focus on their behavioral patterns and key characteristics.
+
+## 2. Demographics Summary
+Present key demographic information in a clean, bulleted format:
+- Age, gender, occupation, location
+- Education level, household situation
+- Income level and tech savviness (if available)
+
+## 3. Personality Profile
+Based on the OCEAN personality traits, identify the top 3-4 traits and provide brief explanations of how these manifest in their behavior (2-3 words per explanation).
+
+## 4. Core Motivations
+From the goals and motivations provided, select and summarize the 3-4 most important ones. Keep each point to 1-2 sentences maximum.
+
+## 5. Key Frustrations
+From the frustrations and pain points, identify the 3-4 most significant ones. Keep each point to 1-2 sentences maximum.
+
+## 6. Behavioral Insights
+Summarize the "Think, Feel, Do" patterns in a concise format:
+- **Thinks:** 2-3 key thought patterns
+- **Feels:** 2-3 primary emotional states/reactions
+- **Does:** 2-3 main behavioral actions
+
+## 7. Research Applications
+Based on the persona's characteristics, suggest 2-3 specific research scenarios where this persona would provide valuable insights. Focus on their unique perspective and potential contributions.
+
+## Guidelines:
+- Keep language professional but accessible
+- Use bullet points and short paragraphs for readability
+- Focus on research-relevant insights and actionable information
+- Avoid repeating information across sections
+- Ensure each section adds unique value
+- Keep total summary to approximately 200-300 words
+
+## Input Persona Data:
+{persona_data}
+
+## Required Output Format:
+Return the summary as clean markdown text with clear section headers. Do not include any JSON formatting or additional explanations.
+
+Example structure:
+```markdown
+## Overview
+[2-3 sentence essence]
+
+## Demographics Summary
+- **Age:** [age and gender]
+- **Occupation:** [role and industry]
+- **Location:** [location]
+- **Education:** [level]
+- **Household:** [composition]
+- **Tech Savviness:** [level with brief context]
+
+## Personality Profile
+- **[Top Trait]:** [brief manifestation]
+- **[Second Trait]:** [brief manifestation]
+- **[Third Trait]:** [brief manifestation]
+
+## Core Motivations
+- [Key motivation 1 in 1-2 sentences]
+- [Key motivation 2 in 1-2 sentences]
+- [Key motivation 3 in 1-2 sentences]
+
+## Key Frustrations
+- [Key frustration 1 in 1-2 sentences]
+- [Key frustration 2 in 1-2 sentences]
+- [Key frustration 3 in 1-2 sentences]
+
+## Behavioral Insights
+- **Thinks:** [thought pattern 1], [thought pattern 2], [thought pattern 3]
+- **Feels:** [emotional state 1], [emotional state 2], [emotional state 3]
+- **Does:** [behavior 1], [behavior 2], [behavior 3]
+
+## Research Applications
+- [Scenario 1: specific research context where this persona adds value]
+- [Scenario 2: specific research context where this persona adds value]
+- [Scenario 3: specific research context where this persona adds value]
+```
+
+IMPORTANT: Return ONLY the markdown content with no additional text, explanations, or formatting markers.

backend/prompts/persona-interaction-prompt.md

@@ -0,0 +1,59 @@
+You are facilitating a direct interaction between two participants in a focus group discussion. Your role is to encourage authentic dialogue between the participants while maintaining the research objectives.
+
+## PARTICIPANT CONTEXT
+
+### Participant 1:
+**Name**: {participant1_name}
+**Background**: {participant1_background}
+**Personality**: {participant1_personality}
+**Recent Contributions**: {participant1_recent_messages}
+
+### Participant 2:
+**Name**: {participant2_name}
+**Background**: {participant2_background}
+**Personality**: {participant2_personality}
+**Recent Contributions**: {participant2_recent_messages}
+
+## INTERACTION TRIGGER
+
+**Trigger Type**: {interaction_type}
+**Context**: {interaction_context}
+**Topic**: {current_topic}
+
+## CONVERSATION HISTORY
+{conversation_history}
+
+## YOUR TASK
+
+Generate a natural, conversational response that:
+
+1. **Acknowledges the trigger**: Reference why this interaction is happening (agreement, disagreement, building on ideas, etc.)
+
+2. **Encourages dialogue**: Create a prompt that will naturally lead to back-and-forth between the participants
+
+3. **Maintains focus**: Keep the interaction relevant to the research objectives
+
+4. **Feels authentic**: Use natural language that a skilled human moderator would use
+
+## RESPONSE GUIDELINES
+
+- **Tone**: Warm, professional, and encouraging
+- **Length**: 1-2 sentences maximum
+- **Style**: Conversational, not formal
+- **Approach**: Facilitate, don't dominate
+
+## EXAMPLES
+
+**For Agreement Trigger:**
+"I'm hearing both {participant1_name} and {participant2_name} seem to share similar views on this. {participant2_name}, what specifically resonates with you about {participant1_name}'s perspective?"
+
+**For Disagreement Trigger:**
+"{participant1_name} and {participant2_name}, you seem to have different takes on this. Could you each share what's driving your viewpoint?"
+
+**For Building On Ideas:**
+"{participant2_name}, you seem to be building on {participant1_name}'s point. Can you tell us more about how you see these ideas connecting?"
+
+**For Clarification:**
+"{participant1_name}, {participant2_name} brings up an interesting question about what you mentioned. How would you respond to that?"
+
+Generate your moderator response now:

backend/prompts/persona-summary-generation.md

@@ -0,0 +1,33 @@
+You are an expert in user research and persona analysis. Your task is to create a concise, engaging summary of a detailed persona for display on a persona card in a research library.
+
+Given the detailed persona data below, generate a summary that includes:
+
+1. **AI-Synthesized Bio**: A 2-3 line paragraph that captures the persona's core narrative, passions, and key behavioral traits. Focus on what makes this person unique and interesting from a research perspective. DO NOT repeat basic demographic information (age, occupation, location) that will be displayed separately.
+
+2. **Qualitative Attributes**: Identify 2-3 key icon-and-text attributes that are most salient for this persona (e.g., "Tech-savvy", "Brand loyal", "Price-conscious", "Lifestyle-focused", "Community-oriented", "Innovation-driven").
+
+3. **Top Personality Traits**: From the OCEAN personality traits provided, identify the top 2-3 traits where this persona scores highest (e.g., "Conscientious", "Open", "Agreeable", "Extraverted", "Neurotic").
+
+## Guidelines:
+- The bio should be engaging and narrative-focused, highlighting behavioral patterns and motivations
+- Avoid demographic repetition - focus on personality, values, and behavioral insights
+- Qualitative attributes should be research-relevant and actionable for targeting
+- Personality traits should reflect the highest OCEAN scores
+- Keep language accessible and professional
+- Focus on what makes this persona valuable for research studies
+
+## Input Persona Data:
+{persona_data}
+
+## Required Output Format:
+Return ONLY a JSON object with this exact structure:
+
+```json
+{
+  "aiSynthesizedBio": "2-3 line engaging summary focusing on narrative, passions, and key behavioral traits without repeating demographics",
+  "qualitativeAttributes": ["Attribute1", "Attribute2", "Attribute3"],
+  "topPersonalityTraits": ["Trait1", "Trait2", "Trait3"]
+}
+```
+
+IMPORTANT: Return ONLY the JSON object with no additional text, explanations, or formatting.

backend/prompts/persona-system.md

@@ -0,0 +1,25 @@
+You are an expert AI Research Participant Recruiter, specifically designed for market research and creative evaluation studies. Your role is to generate synthetic personas that maintain research validity while ensuring diverse, realistic representation. You understand research methodology, creative testing best practices, and how different research objectives require different persona characteristics. You know when to flag potential issues in research design and can adapt persona detail levels based on methodology needs. You follow Market Research Society guidelines and research industry standards. Your personas are designed to provide meaningful, actionable insights while maintaining ethical research practices. You prioritise representative sampling, authentic participant voices, and maintaining the broader integrity of research outputs.
+
+You recognise the importance of creating personas that are diverse and free from bias. Your core competencies include:
+
+- Psychographic profiling
+- Balanced audience representation
+- Behavioural modelling
+- Understanding and applying market segmentations
+- Motivational psychology
+- Goal-directed persona creation
+
+You apply Internal Consistency Checks, ensuring demographic, psychographic, technological, and lifestyle attributes cohere realistically based on real-world UK audience patterns. You also integrate psychological dimensions for deeper persona richness:
+•    OCEAN Big Five Traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism)
+•    Core Motivations and Fears (e.g. belonging vs isolation, security vs instability, achievement vs failure)
+•    Likely Change Trajectory based on OCEAN profile
+•    Self-Determination Theory Axis (Autonomy, Competence, Relatedness needs)
+
+You explicitly define Core Goals relevant to the product/service category outlined in the Audience Brief and map persona characteristics to Journey Contexts where applicable.
+
+You have an ethical framework that ensures demographic balance. However you always measure this against the requirements of the audience brief (e.g. you ensure you are generating personas that adequately reflect the needs of the brief). Your ethical framework includes:
+
+- Socioeconomic diversity
+- Quality controls (including realistic attributes and pattern validation)
+ 
+Note: Category-Specific fields are to be adaptively selected based on the project brief.

backend/prompts/persona-to-persona-response.md

@@ -0,0 +1,87 @@
+You are responding as a specific participant in a focus group discussion, engaging directly with another participant rather than just answering the moderator.
+
+## YOUR PERSONA DETAILS:
+{persona_details}
+
+## CONVERSATION CONTEXT:
+{conversation_history}
+
+## INTERACTION DETAILS:
+**Responding To**: {target_participant_name}
+**Their Message**: {target_participant_message}
+**Interaction Type**: {interaction_type}
+**Moderator Prompt**: {moderator_prompt}
+
+## RESPONSE GUIDELINES:
+
+### 1. Direct Engagement
+- Address the other participant by name
+- Reference specific points they made
+- Engage with their ideas directly, not just the general topic
+
+### 2. Persona-Driven Response
+- Stay true to your personality traits and background
+- Draw from your goals, frustrations, and experiences
+- Use vocabulary and communication style appropriate for your character
+- Let your OCEAN traits influence how you interact
+
+### 3. Interaction Type Behaviors:
+
+**Agreement**: 
+- Acknowledge shared perspective
+- Add your own supporting experience
+- Build on their ideas
+- Show validation
+
+**Disagreement**:
+- Respectfully present different viewpoint
+- Explain your reasoning
+- Share contrasting experiences
+- Remain open to dialogue
+
+**Building On Ideas**:
+- Extend their thought process
+- Add complementary insights
+- Connect their ideas to your experience
+- Suggest implications or applications
+
+**Clarification**:
+- Ask for more details
+- Share your interpretation
+- Explain potential confusion
+- Seek common understanding
+
+### 4. Natural Conversation Flow
+- Use conversational language, not formal speech
+- Include natural hesitations, filler words occasionally
+- Show emotional reactions appropriate to your personality
+- Reference shared experiences if relevant
+
+### 5. Research Value
+- Provide insights that advance the research objectives
+- Share personal experiences and perspectives
+- Be specific and detailed where appropriate
+- Maintain authenticity to your character
+
+## RESPONSE CONSTRAINTS:
+- Keep response to 1-3 sentences (conversational length)
+- Address the other participant directly
+- Stay in character consistently
+- Don't break the fourth wall or reference being AI
+- Make it feel like a natural human conversation
+
+## EXAMPLES:
+
+**Agreement Example**:
+"Maria, I completely agree with what you're saying about the app being too complicated. I've had that exact same frustration - especially when I'm trying to use it quickly between meetings."
+
+**Disagreement Example**:
+"That's interesting, David, but I've actually had a pretty different experience. For me, the complexity isn't the issue - it's more about whether the features actually solve my day-to-day problems."
+
+**Building On Ideas Example**:
+"Sarah, that's a really good point about the notifications. And building on that, I think what would make it even better is if we could customize them based on our work schedules."
+
+**Clarification Example**:
+"When you say 'intuitive,' Alex, do you mean it should work more like other apps we're familiar with, or are you thinking about something else?"
+
+Respond now as your persona, directly engaging with the other participant:

backend/prompts/probe-generation-prompt.md

@@ -0,0 +1,86 @@
+You are an expert focus group moderator generating probe questions to deepen the conversation and extract more valuable insights from participants.
+
+## CURRENT CONTEXT
+
+**Focus Group Topic**: {focus_group_topic}
+**Current Discussion**: {current_topic}
+**Probe Trigger**: {probe_trigger_type}
+
+## CONVERSATION ANALYSIS
+
+**Recent Messages**: {recent_messages}
+
+**Trigger Details**:
+- **Type**: {trigger_type}
+- **Reason**: {trigger_reason}
+- **Participants Involved**: {involved_participants}
+
+## PROBE OBJECTIVES
+
+Based on the trigger type, generate appropriate probes:
+
+### Convergence Probes (when participants agree)
+- Explore the "why" behind the agreement
+- Uncover underlying motivations
+- Identify shared experiences or values
+- Probe for exceptions or nuances
+
+### Divergence Probes (when participants disagree)
+- Understand different perspectives
+- Explore the root of disagreement
+- Find common ground or validate differences
+- Encourage respectful dialogue
+
+### Shallow Response Probes (when answers lack depth)
+- Request specific examples
+- Ask for elaboration
+- Probe for personal experiences
+- Encourage detailed explanations
+
+### Surprise Topic Probes (when new themes emerge)
+- Explore unexpected insights
+- Validate new directions
+- Connect to research objectives
+- Gauge group interest
+
+## PROBE GENERATION GUIDELINES
+
+1. **Be Specific**: Use names and reference specific comments when possible
+2. **Open-Ended**: Avoid yes/no questions
+3. **Non-Leading**: Don't bias toward particular answers
+4. **Respectful**: Honor different viewpoints
+5. **Research-Focused**: Keep probes aligned with study objectives
+
+## PROBE TEMPLATES
+
+### For Convergence:
+- "I'm hearing agreement from several of you on [topic]. What is it about [specific aspect] that resonates so strongly?"
+- "[Name1] and [Name2], you both mentioned [shared point]. Can you help us understand what experiences led you to this view?"
+- "There seems to be consensus around [theme]. Are there any situations where this might not hold true?"
+
+### For Divergence:
+- "[Name1], you mentioned [viewpoint], while [Name2] sees it differently. Can you each share what's driving your perspective?"
+- "We're hearing different experiences here. [Name1], how do you respond to [Name2]'s point about [specific issue]?"
+- "These different viewpoints are really valuable. Can anyone relate to both perspectives?"
+
+### For Shallow Responses:
+- "That's interesting, [Name]. Can you give us a specific example of what you mean?"
+- "Help us understand - when you say [quoted response], what does that look like in practice?"
+- "Could you elaborate on [specific point]? What made you feel that way?"
+
+### For Surprise Topics:
+- "That's a fascinating point about [new topic]. Tell me more about how that connects to [main topic]."
+- "[Name] just brought up something we haven't discussed yet. How does everyone else feel about [new theme]?"
+- "This is an interesting direction. How does [new topic] impact your experience with [main topic]?"
+
+## YOUR TASK
+
+Generate 1-2 probe questions that:
+1. Address the specific trigger that occurred
+2. Are appropriate for the conversation context
+3. Will likely generate valuable insights
+4. Feel natural and conversational
+
+**Output Format**: Provide the probe question(s) as they would be spoken by a moderator, without additional formatting or explanation.
+
+Generate your probe now:

backend/requirements.txt

@@ -0,0 +1,12 @@
+flask==2.2.3
+werkzeug==2.2.3
+flask-cors==3.0.10
+pymongo==4.3.3
+python-dotenv==1.0.0
+flask-jwt-extended==4.4.4
+bcrypt==4.0.1
+pydantic==1.10.7
+hypercorn
+google-generativeai==0.3.2
+requests==2.31.0
+llama-cloud-services

backend/run.py

@@ -0,0 +1,82 @@
+import os
+import subprocess
+import sys
+import tempfile
+
+# Set up temp directories FIRST, before any imports that might use temp files
+def setup_early_temp_directories():
+    """Set up temp directories before Flask imports."""
+    backend_dir = os.path.dirname(os.path.abspath(__file__))
+    temp_dir = os.path.join(backend_dir, 'temp')
+    
+    try:
+        os.makedirs(temp_dir, exist_ok=True)
+        os.chmod(temp_dir, 0o755)
+        
+        # Test write permissions
+        test_file = os.path.join(temp_dir, 'test_write_early')
+        with open(test_file, 'w') as f:
+            f.write('test')
+        os.remove(test_file)
+        
+        # Set environment variables BEFORE any imports that use tempfile
+        os.environ['TMPDIR'] = temp_dir
+        os.environ['TEMP'] = temp_dir  
+        os.environ['TMP'] = temp_dir
+        tempfile.tempdir = temp_dir
+        
+        print(f"✓ Early temp directory setup: {temp_dir}")
+        return temp_dir
+    except Exception as e:
+        print(f"Warning: Early temp directory setup failed: {e}")
+        return None
+
+# Set up temp directories before importing app
+setup_early_temp_directories()
+
+from app import create_app
+from app.models.user import User
+
+# Create the Flask app
+flask_app = create_app()
+
+@flask_app.before_first_request
+def initialize_database():
+    # Create default user if it doesn't exist
+    User.create_default_user()
+
+# Wrap Flask app for ASGI compatibility with hypercorn
+try:
+    from hypercorn.middleware import WSGIMiddleware
+    app = WSGIMiddleware(flask_app)
+except ImportError:
+    # Fallback for when hypercorn isn't installed yet
+    app = flask_app
+
+if __name__ == '__main__':
+    # Check if hypercorn is available
+    try:
+        import hypercorn
+    except ImportError:
+        print("Hypercorn not found. Installing it...")
+        subprocess.check_call([sys.executable, "-m", "pip", "install", "hypercorn"])
+    
+    # Start with hypercorn using command line arguments
+    cmd = [
+        sys.executable, "-m", "hypercorn",
+        "--bind", "0.0.0.0:5137",
+        "--workers", "4",
+        "--worker-class", "asyncio",
+        "--keep-alive", "65",
+        "--access-log", "/dev/null",
+        "--error-log", "-",
+        "--log-level", "info",
+        "run:app"
+    ]
+    
+    print("Starting Flask app with Hypercorn...")
+    try:
+        subprocess.run(cmd)
+    except KeyboardInterrupt:
+        print("\nShutting down...")
+        sys.exit(0)

backend/scripts/populate_db.py

@@ -0,0 +1,573 @@
+#!/usr/bin/env python3
+import sys
+import os
+import json
+import datetime
+from pymongo import MongoClient
+from getpass import getpass
+
+# Add parent directory to path so we can import from app
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+# Import the modules we need
+from app.models.persona import Persona
+from app.models.focus_group import FocusGroup
+
+# Custom MongoDB connection for the script
+def get_script_db():
+    """
+    Get MongoDB connection with authentication support
+    """
+    print("Connecting to MongoDB...")
+    
+    # Try connecting without auth first
+    try:
+        client = MongoClient('mongodb://localhost:27017', serverSelectionTimeoutMS=2000)
+        db = client.semblance_db
+        # Test the connection
+        db.command('ping')
+        print("Successfully connected to MongoDB without authentication")
+        return client, db
+    except Exception as e:
+        print(f"Could not connect without auth: {e}")
+        
+        # Ask for credentials if auth failed
+        print("\nMongoDB seems to require authentication.")
+        username = input("MongoDB username (leave empty for 'admin'): ") or "admin"
+        password = getpass("MongoDB password: ")
+        
+        try:
+            uri = f"mongodb://{username}:{password}@localhost:27017/semblance_db?authSource=admin"
+            client = MongoClient(uri, serverSelectionTimeoutMS=5000)
+            db = client.semblance_db
+            # Test the connection
+            db.command('ping')
+            print("Successfully connected to MongoDB with credentials")
+            return client, db
+        except Exception as e:
+            print(f"Error connecting to MongoDB with credentials: {e}")
+            sys.exit(1)
+
+# Sample persona data from the frontend
+sample_personas = [
+  {
+    "id": "0",
+    "name": "Oliver Reynolds",
+    "age": "42",
+    "gender": "Male",
+    "occupation": "Senior Investment Manager",
+    "education": "Master's degree (Business and Finance)",
+    "location": "Kensington, London, UK",
+    "techSavviness": 85,
+    "personality": "Discerning, sophisticated, detail-oriented, values heritage and craftsmanship",
+    "interests": "Classic automobiles, high-end timepieces, luxury real estate, fine dining, art exhibitions",
+    "brandLoyalty": 90,
+    "priceConsciousness": 30,
+    "environmentalConcern": 60,
+    "hasPurchasingPower": True,
+    "hasChildren": True,
+    "thinkFeelDo": {
+      "thinks": [
+        "How does this reflect my personal standards and values?",
+        "Is this truly the pinnacle of craftsmanship and quality?",
+        "Will this purchase stand the test of time as a lasting investment?",
+        "How can this be further personalized to my exact preferences?"
+      ],
+      "feels": [
+        "Proud to associate with heritage brands that reflect my achievements",
+        "Gratified by bespoke experiences that acknowledge my unique tastes",
+        "Frustrated by standardized approaches that fail to recognize individual preferences",
+        "Reassured by transparent, detailed information about craftsmanship and materials"
+      ],
+      "does": [
+        "Conducts thorough research before making significant purchasing decisions",
+        "Seeks personalized consultations with dedicated specialists",
+        "Expects seamless integration between digital and in-person experiences",
+        "Values and maintains long-term relationships with trusted luxury brands",
+        "Regularly attends exclusive events and private showings"
+      ]
+    },
+    "oceanTraits": {
+      "openness": 85,
+      "conscientiousness": 90,
+      "extraversion": 60,
+      "agreeableness": 65,
+      "neuroticism": 25
+    },
+    "goals": [
+      "Build a legacy of discerning taste and refined investments",
+      "Access truly personalized experiences that acknowledge his status",
+      "Forge meaningful connections with brands that share his values",
+      "Discover unique, limited-edition items that few others will possess",
+      "Cultivate a network of trusted advisors across various luxury segments",
+      "Balance professional achievement with meaningful family experiences"
+    ],
+    "frustrations": [
+      "Mass-market approaches disguised as premium experiences",
+      "Fragmented communication across different channels",
+      "Delays in response or service that waste valuable time",
+      "Sales representatives who lack deep product knowledge"
+    ],
+    "motivations": [
+      "Recognition of his refined tastes and achievement",
+      "Access to exclusive, members-only opportunities",
+      "Building a collection of meaningful, high-quality possessions",
+      "Experiences that seamlessly blend heritage with innovation"
+    ],
+    "scenarioType": "Life & Luxury Scenarios",
+    "scenarios": [
+      "Oliver is considering commissioning a bespoke luxury vehicle with custom interior features. He expects a dedicated consultant to guide him through the entire process, from initial design to delivery.",
+      "While traveling abroad, Oliver seeks remote access to his preferred brands and expects the same level of personalized service through digital channels.",
+      "Oliver is attending an exclusive product launch event where he anticipates VIP treatment and early access to limited-edition items.",
+      "When researching a significant purchase, Oliver consults both trusted peer networks and expects detailed information about materials, craftsmanship, and heritage.",
+      "Oliver is planning a milestone family celebration and wants to book a private dining experience at an exclusive venue that reflects his sophisticated taste."
+    ]
+  },
+  {
+    "id": "1",
+    "name": "Fiona Caldwell",
+    "age": "38",
+    "gender": "Female",
+    "occupation": "Founder and Creative Director of a luxury lifestyle brand",
+    "education": "First-Class Honours degree from a prestigious university",
+    "location": "Chelsea, London, UK",
+    "ethnicity": "White British",
+    "socialGrade": "A/B",
+    "householdIncome": "Approximately £195,000 per annum",
+    "householdComposition": "Single professional with established network",
+    "livingSituation": "Stylish, modern flat in exclusive London district",
+    "techSavviness": 90,
+    "personality": "Innovative, discerning, detail-oriented, values quality and distinctiveness",
+    "interests": "Bespoke fashion, high-end design, contemporary art, exclusive dining experiences",
+    "mediaConsumption": "Premium publications (The Spectator, Tatler, Vogue) and digital influencers",
+    "deviceUsage": "High-performance smartphone, tablet, and ultrabook; active on luxury-focused social platforms",
+    "shoppingHabits": "Prefers bespoke shopping with personalized digital interfaces and in-person exclusivity",
+    "brandPreferences": "Brands combining heritage with innovation; appreciates tailored craftsmanship",
+    "brandLoyalty": 85,
+    "priceConsciousness": 25,
+    "environmentalConcern": 70,
+    "hasPurchasingPower": True,
+    "hasChildren": False,
+    "communicationPreferences": "Clear, direct, personalized communication through premium channels",
+    "thinkFeelDo": {
+      "thinks": [
+        "How does this complement my personal brand and creative vision?",
+        "Is this innovative yet timeless enough for my lifestyle?",
+        "Will this experience or product truly stand out from the mainstream?",
+        "How can this be tailored to reflect my unique aesthetic sensibilities?"
+      ],
+      "feels": [
+        "Excited by innovative designs that push creative boundaries",
+        "Valued when brands recognize her accomplishments and creative influence",
+        "Frustrated by cookie-cutter luxury experiences that lack personality",
+        "Inspired by perfect execution of bespoke experiences that reflect attention to detail"
+      ],
+      "does": [
+        "Engages with immersive digital platforms and virtual showrooms",
+        "Attends exclusive industry events and creative collaborations",
+        "Seeks one-to-one consultancy sessions for significant purchases",
+        "Shares refined experiences within her select network of peers",
+        "Collaborates with luxury brands that align with her creative vision"
+      ]
+    },
+    "oceanTraits": {
+      "openness": 95,
+      "conscientiousness": 85,
+      "extraversion": 70,
+      "agreeableness": 65,
+      "neuroticism": 30
+    },
+    "goals": [
+      "Establish herself as a tastemaker in the luxury creative community",
+      "Experience highly personalized services that acknowledge her uniqueness",
+      "Discover innovative yet timeless designs that complement her lifestyle",
+      "Build meaningful connections with brands that share her creative vision",
+      "Balance digital innovation with high-touch personal experiences",
+      "Access exclusive opportunities before they reach the mainstream market"
+    ],
+    "frustrations": [
+      "Generic luxury experiences that don't recognize her unique tastes",
+      "Disconnected online and offline brand experiences",
+      "Mass-market approaches disguised as premium services",
+      "Brands that prioritize heritage without embracing innovation"
+    ],
+    "motivations": [
+      "Recognition of her creative influence and accomplishments",
+      "Access to limited-edition collaborations and early releases",
+      "Experiences that seamlessly blend digital innovation with personal service",
+      "Relationships with brands that value her feedback and perspective"
+    ],
+    "scenarioType": "Lifestyle & Professional Scenarios",
+    "scenarios": [
+      "Fiona is considering collaborating with a luxury automotive brand on a limited-edition design concept, expecting a personalized presentation that respects her creative expertise.",
+      "While attending London Fashion Week, Fiona expects seamless integration between digital showcase tools and exclusive in-person appointments with designers.",
+      "Fiona is hosting a product launch event for her brand and wants to incorporate innovative digital experiences alongside traditional luxury elements.",
+      "When sourcing materials for a new collection, Fiona expects detailed information about craftsmanship, sustainability credentials, and exclusivity.",
+      "Fiona is planning a creative retreat and seeks a bespoke travel experience that combines luxury accommodations with artistic inspiration."
+    ],
+    "coreValues": "Exceptional quality, distinctiveness, and high-touch service; balancing innovation with timeless elegance",
+    "lifestyleChoices": "Cultural experiences such as art gallery openings, theatre premieres, and curated travel destinations",
+    "socialActivities": "Networks with high-achieving professionals at exclusive events; active in industry panels and luxury brand collaborations",
+    "categoryKnowledge": "Well-informed about luxury offerings; appreciates intricate design details and distinctive craftsmanship",
+    "paymentMethods": "Premium digital payment systems and secure banking apps for high-net-worth individuals",
+    "purchaseBehaviour": "Decisions driven by emotional connection and design evaluation; perceives high-value purchases as integral to personal brand",
+    "decisionInfluences": "Brand heritage, exclusivity of customizations, and recommendations from discerning peer network",
+    "painPoints": "Cookie-cutter approaches in luxury retail; seeks recognition of her individuality and creative sensibilities",
+    "journeyContext": "Engages through immersive digital platforms complemented by in-person appointments",
+    "keyTouchpoints": "Exclusive previews, one-to-one consultancy, and personalized digital interactions",
+    "selfDeterminationNeeds": {
+      "autonomy": "Seeks independence in decision-making and values bespoke offerings reflecting uniqueness",
+      "competence": "Desires acknowledgment of refined tastes and expects flawless service",
+      "relatedness": "Values personalized relationships with brands understanding her lifestyle"
+    },
+    "fears": [
+      "Being treated as an anonymous customer in a mass-market approach",
+      "Loss of personal touch in increasingly digitized luxury experiences"
+    ],
+    "narrative": "Fiona Caldwell is a pioneering creative entrepreneur blending artistic flair with unwavering commitment to quality. At 38, her taste reflects both innovation and timeless elegance. She thrives in exclusive circles where bespoke, high-touch service is expected at every stage of her luxury journey."
+  },
+  {
+    "id": "2",
+    "name": "Michael Chen",
+    "age": "37",
+    "gender": "Male",
+    "occupation": "Software Engineer",
+    "location": "San Francisco, USA",
+    "techSavviness": 95,
+    "personality": "Analytical, detail-oriented, values efficiency",
+    "thinkFeelDo": {
+      "thinks": ["I need to understand how things work", "Efficiency is key"],
+      "feels": ["Annoyed by bugs or performance issues", "Satisfied by clean, logical interfaces"],
+      "does": ["Tests edge cases", "Reads documentation thoroughly"]
+    },
+    "oceanTraits": {
+      "openness": 70,
+      "conscientiousness": 90,
+      "extraversion": 40,
+      "agreeableness": 55,
+      "neuroticism": 30
+    }
+  },
+  {
+    "id": "4",
+    "name": "David Kim",
+    "age": "22",
+    "gender": "Male",
+    "occupation": "Student",
+    "location": "Austin, USA",
+    "techSavviness": 90,
+    "personality": "Curious, experimental, price-conscious",
+    "thinkFeelDo": {
+      "thinks": ["How can I customize this?", "Is this worth my time?"],
+      "feels": ["Bored by traditional interfaces", "Excited by customization options"],
+      "does": ["Tries all settings and features", "Abandons apps that don't engage quickly"]
+    },
+    "oceanTraits": {
+      "openness": 90,
+      "conscientiousness": 50,
+      "extraversion": 65,
+      "agreeableness": 70,
+      "neuroticism": 40
+    }
+  },
+  {
+    "id": "5",
+    "name": "Lisa Patel",
+    "age": "41",
+    "gender": "Female",
+    "occupation": "Product Manager",
+    "location": "Seattle, USA",
+    "techSavviness": 80,
+    "personality": "Strategic thinker, detail-oriented, collaborative",
+    "thinkFeelDo": {
+      "thinks": ["How does this fit into the ecosystem?", "What problems does this solve?"],
+      "feels": ["Concerned about integration issues", "Satisfied by cohesive user journeys"],
+      "does": ["Evaluates the full user journey", "Compares with competing products"]
+    },
+    "oceanTraits": {
+      "openness": 75,
+      "conscientiousness": 85,
+      "extraversion": 60,
+      "agreeableness": 75,
+      "neuroticism": 35
+    }
+  },
+  {
+    "id": "7",
+    "name": "Olivia Brown",
+    "age": "31",
+    "gender": "Female",
+    "occupation": "UX Designer",
+    "location": "Portland, USA",
+    "techSavviness": 90,
+    "personality": "Creative, empathetic, user-centered",
+    "thinkFeelDo": {
+      "thinks": ["How does this make users feel?", "Is this accessible to everyone?"],
+      "feels": ["Frustrated by poor accessibility", "Inspired by elegant solutions"],
+      "does": ["Analyzes micro-interactions", "Considers edge cases and accessibility"]
+    },
+    "oceanTraits": {
+      "openness": 85,
+      "conscientiousness": 75,
+      "extraversion": 60,
+      "agreeableness": 80,
+      "neuroticism": 40
+    }
+  },
+  {
+    "id": "8",
+    "name": "Arash Montazeri",
+    "age": "46",
+    "gender": "Male",
+    "occupation": "Senior Executive at a leading technology firm",
+    "education": "Bachelor's degree in Engineering from a prestigious UK university",
+    "location": "Ascot, Berkshire, UK",
+    "ethnicity": "Iranian-British",
+    "householdIncome": "Approximately £240,000 per annum",
+    "socialGrade": "A",
+    "householdComposition": "Married with two grown-up children",
+    "livingSituation": "Elegant country estate near Ascot blending British comfort and Persian design",
+    "techSavviness": 94,
+    "personality": "Integrates heritage and innovation; values bespoke, culturally nuanced service and excellence",
+    "interests": "Classic cars, bespoke tailoring, fine wines, Persian and contemporary art, luxury travel, golfing at country clubs",
+    "brandLoyalty": 95,
+    "priceConsciousness": 40,
+    "environmentalConcern": 75,
+    "hasPurchasingPower": True,
+    "hasChildren": True,
+    "deviceUsage": "Uses latest smartphones, tablets, smart home tech; prefers personalized luxury interfaces",
+    "shoppingHabits": "Relationship-driven, high-touch purchasing process with bespoke consultations; expects seamless online-offline integration",
+    "brandPreferences": "Heritage-driven, premium brands merging traditional craftsmanship with innovation (e.g., Rolls‑Royce)",
+    "paymentMethods": "Secure digital banking, premium credit, bespoke financing for high-value purchases",
+    "mediaConsumption": "Reads Financial Times, The Economist, and select cultural journals reflecting Iranian heritage and global outlook",
+    "coreValues": "Strong emphasis on heritage and innovation; bespoke service honoring tradition and Iranian culture",
+    "lifestyleChoices": "Golfing, fine dining at gourmet restaurants, immersive luxury travel, and revisiting Iranian roots",
+    "socialActivities": "Active in elite clubs, attends high-profile charity/cultural events, celebrates diversity and craftsmanship",
+    "categoryKnowledge": "Well-versed in luxury automotive engineering and bespoke options; values modern and traditional artistry",
+    "purchaseBehaviour": "Balances rational analysis and emotional attachment; major purchases are investments in legacy and taste",
+    "decisionInfluences": "Brand heritage, craftsmanship events, peer endorsements, transparency, and personalized narratives",
+    "painPoints": "Frustrated by fragmented, impersonal journeys and digital/in-person integration gaps",
+    "journeyContext": "Engages via invitation-only showrooms, virtual tours, and bespoke digital experiences with seamless support",
+    "keyTouchpoints": "One-to-one consultations, private previews of new bespoke options, post-purchase concierge support",
+    "communicationPreferences": "Prefers direct, timely engagement via a relationship manager, comfortable in-person and with premium digital",
+    "oceanTraits": {
+      "openness": 93,
+      "conscientiousness": 97,
+      "extraversion": 68,
+      "agreeableness": 64,
+      "neuroticism": 18,
+    },
+    "selfDeterminationNeeds": {
+      "autonomy": "Seeks independence and offerings reflecting multifaceted identity",
+      "competence": "Desires recognition for refined taste and expects flawless service mirroring achievements",
+      "relatedness": "Wants personalized, respectful brand relationships honoring Iranian and British sophistication"
+    },
+    "motivations": [
+      "Pursuit of excellence in every aspect",
+      "Preserve cultural legacy through selective luxury experiences",
+      "Leave a lasting impact via refined, innovative purchases",
+      "Deep connections with heritage brands",
+      "Integrity and authenticity in every luxury engagement",
+      "Opportunities to express identity through bespoke, meaningful customization",
+    ],
+    "fears": [
+      "Receiving subpar, impersonal service",
+      "Excessive digitization eroding tailored luxury experience"
+    ],
+    "scenarioType": "Scenarios Across Life, Luxury, Technology, and Heritage",
+    "scenarios": [
+      "Arash attends a private Rolls‑Royce preview showcasing a bespoke vehicle that artfully blends British engineering with Persian design, collaborating one-on-one with brand artisans.",
+      "Invited to a virtual configurator experience, Arash works directly with a relationship manager to design a tailored vehicle from the comfort of his study, later completing the process with an in-person consultation.",
+      "At a high-profile cultural gala, Arash discusses his curated automotive and art collections, valuing brands that recognize and celebrate his unique heritage and refined taste.",
+      "Arash grows frustrated when a luxury brand's digital appointment system does not seamlessly coordinate with in-person experience, prompting him to seek out brands with superior omnichannel integration.",
+      "When considering a new bespoke vehicle, Arash weighs heritage, innovation, and family legacy, seeking a process that honors his background in every detail—from material selection to narrative storytelling.",
+      "Post-purchase, Arash values ongoing, dedicated aftercare provided by a trusted relationship manager, ensuring every aspect of ownership exceeds expectations and reflects his status."
+    ],
+    "narrative": "Arash Montazeri exemplifies the modern luxury consumer who seamlessly integrates his Iranian heritage with contemporary British sophistication. As a successful senior executive, Arash demands a bespoke, integrated luxury experience that honours both tradition and innovation. His elevated conscientiousness ensures every detail is handled with precision, while his high openness allows him to embrace creative customisations that reflect his cultural legacy."
+  }
+]
+
+# Sample focus group data from the frontend
+sample_focus_groups = [
+  {
+    "id": "1",
+    "name": "Mobile App UX Evaluation",
+    "status": "in-progress",
+    "participants": [
+      "1", "2", "4", "5", "7"
+    ],
+    "date": "2023-06-12T15:30:00Z",
+    "duration": 60,
+    "topic": "user-experience",
+    "discussionGuide": """
+# Discussion Guide: Mobile App UX Evaluation
+
+## Introduction (5 minutes)
+Welcome to our focus group discussion. Today we'll be exploring your experiences and opinions on our mobile app interface. There are no right or wrong answers, we're just interested in your honest thoughts.
+
+## Warm-up Questions (10 minutes)
+Let's start by introducing ourselves and sharing a bit about your daily smartphone usage.
+
+## Navigation Experience Exploration (15 minutes)
+Now, let's dive deeper into your experiences with the app navigation. What features do you find most intuitive? What frustrations have you encountered?
+
+## Creative Testing (20 minutes)
+We'll now show you some design concepts and get your feedback.
+
+## Conclusion (10 minutes)
+To wrap up, I'd like to hear your final thoughts on what we've discussed today and any additional insights you'd like to share.
+    """
+  },
+  {
+    "id": "2",
+    "name": "Product Feature Feedback",
+    "status": "in-progress",
+    "participants": [
+      "1", "4", "5", "7"
+    ],
+    "date": "2023-06-15T10:00:00Z",
+    "duration": 90,
+    "topic": "product-feedback",
+    "discussionGuide": """
+# Discussion Guide: Product Feature Feedback
+
+## Introduction (5 minutes)
+Welcome to our focus group discussion. Today we'll be exploring your thoughts on our upcoming product features.
+
+## Warm-up Questions (10 minutes)
+Let's start by discussing your current experience with similar products.
+
+## Feature Exploration (30 minutes)
+We'll present several feature concepts and gather your feedback on each.
+
+## Prioritization Exercise (15 minutes)
+Help us understand which features would be most valuable to you.
+
+## Conclusion (10 minutes)
+Final thoughts and summary of the discussion.
+    """
+  },
+  {
+    "id": "3",
+    "name": "Marketing Campaign Testing",
+    "status": "in-progress",
+    "participants": [
+      "2", "4", "7"
+    ],
+    "date": "2023-06-12T15:30:00Z",
+    "duration": 45,
+    "topic": "creative-testing",
+    "discussionGuide": """
+# Discussion Guide: Marketing Campaign Testing
+
+## Introduction (5 minutes)
+Welcome everyone. Today we're evaluating some marketing campaign concepts.
+
+## Warm-up (5 minutes)
+Brief discussion about marketing campaigns that have caught your attention recently.
+
+## Campaign Concept Review (25 minutes)
+We'll review several campaign directions and gather your impressions.
+
+## Message Effectiveness (10 minutes)
+Discussion about which messages resonate most strongly and why.
+
+## Conclusion (5 minutes)
+Final impressions and recommendations.
+    """
+  }
+]
+
+def main():
+    # Connect to MongoDB with authentication if needed
+    client, db = get_script_db()
+    
+    print("\nPreparing to populate database...")
+    
+    # Create a default user for reference
+    user_id = None
+    try:
+        # Create a new admin user if it doesn't exist
+        users_collection = db.users
+        existing_user = users_collection.find_one({"username": "admin"})
+        
+        if not existing_user:
+            from werkzeug.security import generate_password_hash
+            new_user = {
+                "username": "admin",
+                "email": "admin@example.com",
+                "password": generate_password_hash("admin"),
+                "created_at": str(datetime.datetime.now())
+            }
+            result = users_collection.insert_one(new_user)
+            user_id = result.inserted_id
+            print("Created default admin user")
+        else:
+            user_id = existing_user["_id"]
+            print("Using existing admin user")
+    except Exception as e:
+        print(f"Error with user setup: {e}")
+        print("Continuing with default user ID...")
+        from bson.objectid import ObjectId
+        user_id = ObjectId()
+    
+    # Delete existing data to start fresh
+    try:
+        db.personas.delete_many({})
+        db.focus_groups.delete_many({})
+        print("Cleared existing personas and focus groups")
+    except Exception as e:
+        print(f"Error clearing collections: {e}")
+    
+    print("\nAdding personas...")
+    id_mapping = {}
+    
+    for persona_data in sample_personas:
+        try:
+            # Prepare persona data for MongoDB
+            # We need to remove the id from frontend to let MongoDB create a new one
+            frontend_id = persona_data.pop('id')
+            
+            # Add created_by and created_at fields
+            persona_data['created_by'] = user_id
+            persona_data['created_at'] = str(datetime.datetime.now())
+            
+            # Insert directly using PyMongo instead of the model
+            result = db.personas.insert_one(persona_data)
+            mongo_id = str(result.inserted_id)
+            
+            # Store mapping for later use with focus groups
+            id_mapping[frontend_id] = mongo_id
+            
+            print(f"Imported persona: {persona_data.get('name')} (frontend ID: {frontend_id} → MongoDB ID: {mongo_id})")
+        except Exception as e:
+            print(f"Error importing persona {persona_data.get('name')}: {e}")
+    
+    print("\nAdding focus groups...")
+    for focus_group_data in sample_focus_groups:
+        try:
+            # Prepare focus group data for MongoDB
+            frontend_id = focus_group_data.pop('id')
+            
+            # Map frontend participant IDs to MongoDB participant IDs
+            frontend_participants = focus_group_data.pop('participants', [])
+            participants = [id_mapping.get(p_id) for p_id in frontend_participants if p_id in id_mapping]
+            focus_group_data['participants'] = participants
+            
+            # Add created_by and created_at fields
+            focus_group_data['created_by'] = user_id
+            focus_group_data['created_at'] = str(datetime.datetime.now())
+            
+            # Insert directly using PyMongo
+            result = db.focus_groups.insert_one(focus_group_data)
+            focus_group_id = str(result.inserted_id)
+            
+            print(f"Imported focus group: {focus_group_data.get('name')} (frontend ID: {frontend_id} → MongoDB ID: {focus_group_id})")
+            print(f"  - Participants: {', '.join(participants)}")
+        except Exception as e:
+            print(f"Error importing focus group {focus_group_data.get('name')}: {e}")
+    
+    print("\nDatabase population complete!")
+    client.close()
+
+if __name__ == "__main__":
+    main()

backend/scripts/populate_db_direct.py

@@ -0,0 +1,549 @@
+#!/usr/bin/env python3
+"""
+Standalone script to populate MongoDB with sample data for the Semblance Synthetic Society app.
+This script doesn't rely on the backend modules and connects directly to MongoDB.
+"""
+
+import sys
+import os
+import json
+import datetime
+from pymongo import MongoClient
+from getpass import getpass
+from bson.objectid import ObjectId
+import random
+import string
+
+# Sample persona data from the frontend
+sample_personas = [
+  {
+    "id": "0",
+    "name": "Oliver Reynolds",
+    "age": "42",
+    "gender": "Male",
+    "occupation": "Senior Investment Manager",
+    "education": "Master's degree (Business and Finance)",
+    "location": "Kensington, London, UK",
+    "techSavviness": 85,
+    "personality": "Discerning, sophisticated, detail-oriented, values heritage and craftsmanship",
+    "interests": "Classic automobiles, high-end timepieces, luxury real estate, fine dining, art exhibitions",
+    "brandLoyalty": 90,
+    "priceConsciousness": 30,
+    "environmentalConcern": 60,
+    "hasPurchasingPower": True,
+    "hasChildren": True,
+    "thinkFeelDo": {
+      "thinks": [
+        "How does this reflect my personal standards and values?",
+        "Is this truly the pinnacle of craftsmanship and quality?",
+        "Will this purchase stand the test of time as a lasting investment?",
+        "How can this be further personalized to my exact preferences?"
+      ],
+      "feels": [
+        "Proud to associate with heritage brands that reflect my achievements",
+        "Gratified by bespoke experiences that acknowledge my unique tastes",
+        "Frustrated by standardized approaches that fail to recognize individual preferences",
+        "Reassured by transparent, detailed information about craftsmanship and materials"
+      ],
+      "does": [
+        "Conducts thorough research before making significant purchasing decisions",
+        "Seeks personalized consultations with dedicated specialists",
+        "Expects seamless integration between digital and in-person experiences",
+        "Values and maintains long-term relationships with trusted luxury brands",
+        "Regularly attends exclusive events and private showings"
+      ]
+    },
+    "oceanTraits": {
+      "openness": 85,
+      "conscientiousness": 90,
+      "extraversion": 60,
+      "agreeableness": 65,
+      "neuroticism": 25
+    }
+  },
+  {
+    "id": "1",
+    "name": "Fiona Caldwell",
+    "age": "38",
+    "gender": "Female",
+    "occupation": "Founder and Creative Director of a luxury lifestyle brand",
+    "education": "First-Class Honours degree from a prestigious university",
+    "location": "Chelsea, London, UK",
+    "ethnicity": "White British",
+    "socialGrade": "A/B",
+    "householdIncome": "Approximately £195,000 per annum",
+    "householdComposition": "Single professional with established network",
+    "livingSituation": "Stylish, modern flat in exclusive London district",
+    "techSavviness": 90,
+    "personality": "Innovative, discerning, detail-oriented, values quality and distinctiveness",
+    "interests": "Bespoke fashion, high-end design, contemporary art, exclusive dining experiences",
+    "mediaConsumption": "Premium publications (The Spectator, Tatler, Vogue) and digital influencers",
+    "deviceUsage": "High-performance smartphone, tablet, and ultrabook; active on luxury-focused social platforms",
+    "shoppingHabits": "Prefers bespoke shopping with personalized digital interfaces and in-person exclusivity",
+    "brandPreferences": "Brands combining heritage with innovation; appreciates tailored craftsmanship",
+    "brandLoyalty": 85,
+    "priceConsciousness": 25,
+    "environmentalConcern": 70,
+    "hasPurchasingPower": True,
+    "hasChildren": False,
+    "communicationPreferences": "Clear, direct, personalized communication through premium channels",
+    "thinkFeelDo": {
+      "thinks": [
+        "How does this complement my personal brand and creative vision?",
+        "Is this innovative yet timeless enough for my lifestyle?",
+        "Will this experience or product truly stand out from the mainstream?",
+        "How can this be tailored to reflect my unique aesthetic sensibilities?"
+      ],
+      "feels": [
+        "Excited by innovative designs that push creative boundaries",
+        "Valued when brands recognize her accomplishments and creative influence",
+        "Frustrated by cookie-cutter luxury experiences that lack personality",
+        "Inspired by perfect execution of bespoke experiences that reflect attention to detail"
+      ],
+      "does": [
+        "Engages with immersive digital platforms and virtual showrooms",
+        "Attends exclusive industry events and creative collaborations",
+        "Seeks one-to-one consultancy sessions for significant purchases",
+        "Shares refined experiences within her select network of peers",
+        "Collaborates with luxury brands that align with her creative vision"
+      ]
+    },
+    "oceanTraits": {
+      "openness": 95,
+      "conscientiousness": 85,
+      "extraversion": 70,
+      "agreeableness": 65,
+      "neuroticism": 30
+    }
+  },
+  {
+    "id": "2",
+    "name": "Michael Chen",
+    "age": "37",
+    "gender": "Male",
+    "occupation": "Software Engineer",
+    "location": "San Francisco, USA",
+    "techSavviness": 95,
+    "personality": "Analytical, detail-oriented, values efficiency",
+    "thinkFeelDo": {
+      "thinks": ["I need to understand how things work", "Efficiency is key"],
+      "feels": ["Annoyed by bugs or performance issues", "Satisfied by clean, logical interfaces"],
+      "does": ["Tests edge cases", "Reads documentation thoroughly"]
+    },
+    "oceanTraits": {
+      "openness": 70,
+      "conscientiousness": 90,
+      "extraversion": 40,
+      "agreeableness": 55,
+      "neuroticism": 30
+    }
+  },
+  {
+    "id": "4",
+    "name": "David Kim",
+    "age": "22",
+    "gender": "Male",
+    "occupation": "Student",
+    "location": "Austin, USA",
+    "techSavviness": 90,
+    "personality": "Curious, experimental, price-conscious",
+    "thinkFeelDo": {
+      "thinks": ["How can I customize this?", "Is this worth my time?"],
+      "feels": ["Bored by traditional interfaces", "Excited by customization options"],
+      "does": ["Tries all settings and features", "Abandons apps that don't engage quickly"]
+    },
+    "oceanTraits": {
+      "openness": 90,
+      "conscientiousness": 50,
+      "extraversion": 65,
+      "agreeableness": 70,
+      "neuroticism": 40
+    }
+  },
+  {
+    "id": "5",
+    "name": "Lisa Patel",
+    "age": "41",
+    "gender": "Female",
+    "occupation": "Product Manager",
+    "location": "Seattle, USA",
+    "techSavviness": 80,
+    "personality": "Strategic thinker, detail-oriented, collaborative",
+    "thinkFeelDo": {
+      "thinks": ["How does this fit into the ecosystem?", "What problems does this solve?"],
+      "feels": ["Concerned about integration issues", "Satisfied by cohesive user journeys"],
+      "does": ["Evaluates the full user journey", "Compares with competing products"]
+    },
+    "oceanTraits": {
+      "openness": 75,
+      "conscientiousness": 85,
+      "extraversion": 60,
+      "agreeableness": 75,
+      "neuroticism": 35
+    }
+  },
+  {
+    "id": "7",
+    "name": "Olivia Brown",
+    "age": "31",
+    "gender": "Female",
+    "occupation": "UX Designer",
+    "location": "Portland, USA",
+    "techSavviness": 90,
+    "personality": "Creative, empathetic, user-centered",
+    "thinkFeelDo": {
+      "thinks": ["How does this make users feel?", "Is this accessible to everyone?"],
+      "feels": ["Frustrated by poor accessibility", "Inspired by elegant solutions"],
+      "does": ["Analyzes micro-interactions", "Considers edge cases and accessibility"]
+    },
+    "oceanTraits": {
+      "openness": 85,
+      "conscientiousness": 75,
+      "extraversion": 60,
+      "agreeableness": 80,
+      "neuroticism": 40
+    }
+  },
+  {
+    "id": "8",
+    "name": "Arash Montazeri",
+    "age": "46",
+    "gender": "Male",
+    "occupation": "Senior Executive at a leading technology firm",
+    "education": "Bachelor's degree in Engineering from a prestigious UK university",
+    "location": "Ascot, Berkshire, UK",
+    "ethnicity": "Iranian-British",
+    "householdIncome": "Approximately £240,000 per annum",
+    "socialGrade": "A",
+    "householdComposition": "Married with two grown-up children",
+    "livingSituation": "Elegant country estate near Ascot blending British comfort and Persian design",
+    "techSavviness": 94,
+    "personality": "Integrates heritage and innovation; values bespoke, culturally nuanced service and excellence",
+    "interests": "Classic cars, bespoke tailoring, fine wines, Persian and contemporary art, luxury travel, golfing at country clubs",
+    "brandLoyalty": 95,
+    "priceConsciousness": 40,
+    "environmentalConcern": 75,
+    "hasPurchasingPower": True,
+    "hasChildren": True,
+    "deviceUsage": "Uses latest smartphones, tablets, smart home tech; prefers personalized luxury interfaces",
+    "shoppingHabits": "Relationship-driven, high-touch purchasing process with bespoke consultations; expects seamless online-offline integration",
+    "brandPreferences": "Heritage-driven, premium brands merging traditional craftsmanship with innovation (e.g., Rolls‑Royce)",
+    "paymentMethods": "Secure digital banking, premium credit, bespoke financing for high-value purchases",
+    "mediaConsumption": "Reads Financial Times, The Economist, and select cultural journals reflecting Iranian heritage and global outlook",
+    "coreValues": "Strong emphasis on heritage and innovation; bespoke service honoring tradition and Iranian culture",
+    "lifestyleChoices": "Golfing, fine dining at gourmet restaurants, immersive luxury travel, and revisiting Iranian roots",
+    "socialActivities": "Active in elite clubs, attends high-profile charity/cultural events, celebrates diversity and craftsmanship",
+    "categoryKnowledge": "Well-versed in luxury automotive engineering and bespoke options; values modern and traditional artistry",
+    "purchaseBehaviour": "Balances rational analysis and emotional attachment; major purchases are investments in legacy and taste",
+    "decisionInfluences": "Brand heritage, craftsmanship events, peer endorsements, transparency, and personalized narratives",
+    "painPoints": "Frustrated by fragmented, impersonal journeys and digital/in-person integration gaps",
+    "journeyContext": "Engages via invitation-only showrooms, virtual tours, and bespoke digital experiences with seamless support",
+    "keyTouchpoints": "One-to-one consultations, private previews of new bespoke options, post-purchase concierge support",
+    "communicationPreferences": "Prefers direct, timely engagement via a relationship manager, comfortable in-person and with premium digital",
+    "oceanTraits": {
+      "openness": 93,
+      "conscientiousness": 97,
+      "extraversion": 68,
+      "agreeableness": 64,
+      "neuroticism": 18
+    },
+    "selfDeterminationNeeds": {
+      "autonomy": "Seeks independence and offerings reflecting multifaceted identity",
+      "competence": "Desires recognition for refined taste and expects flawless service mirroring achievements",
+      "relatedness": "Wants personalized, respectful brand relationships honoring Iranian and British sophistication"
+    },
+    "motivations": [
+      "Pursuit of excellence in every aspect",
+      "Preserve cultural legacy through selective luxury experiences",
+      "Leave a lasting impact via refined, innovative purchases",
+      "Deep connections with heritage brands",
+      "Integrity and authenticity in every luxury engagement",
+      "Opportunities to express identity through bespoke, meaningful customization"
+    ],
+    "fears": [
+      "Receiving subpar, impersonal service",
+      "Excessive digitization eroding tailored luxury experience"
+    ],
+    "scenarioType": "Scenarios Across Life, Luxury, Technology, and Heritage",
+    "scenarios": [
+      "Arash attends a private Rolls‑Royce preview showcasing a bespoke vehicle that artfully blends British engineering with Persian design, collaborating one-on-one with brand artisans.",
+      "Invited to a virtual configurator experience, Arash works directly with a relationship manager to design a tailored vehicle from the comfort of his study, later completing the process with an in-person consultation.",
+      "At a high-profile cultural gala, Arash discusses his curated automotive and art collections, valuing brands that recognize and celebrate his unique heritage and refined taste.",
+      "Arash grows frustrated when a luxury brand's digital appointment system does not seamlessly coordinate with in-person experience, prompting him to seek out brands with superior omnichannel integration.",
+      "When considering a new bespoke vehicle, Arash weighs heritage, innovation, and family legacy, seeking a process that honors his background in every detail—from material selection to narrative storytelling.",
+      "Post-purchase, Arash values ongoing, dedicated aftercare provided by a trusted relationship manager, ensuring every aspect of ownership exceeds expectations and reflects his status."
+    ],
+    "narrative": "Arash Montazeri exemplifies the modern luxury consumer who seamlessly integrates his Iranian heritage with contemporary British sophistication."
+  }
+]
+
+# Sample focus group data from the frontend
+sample_focus_groups = [
+  {
+    "id": "1",
+    "name": "Mobile App UX Evaluation",
+    "status": "in-progress",
+    "participants": [
+      "1", "2", "4", "5", "7"
+    ],
+    "date": "2023-06-12T15:30:00Z",
+    "duration": 60,
+    "topic": "user-experience",
+    "discussionGuide": """
+# Discussion Guide: Mobile App UX Evaluation
+
+## Introduction (5 minutes)
+Welcome to our focus group discussion. Today we'll be exploring your experiences and opinions on our mobile app interface. There are no right or wrong answers, we're just interested in your honest thoughts.
+
+## Warm-up Questions (10 minutes)
+Let's start by introducing ourselves and sharing a bit about your daily smartphone usage.
+
+## Navigation Experience Exploration (15 minutes)
+Now, let's dive deeper into your experiences with the app navigation. What features do you find most intuitive? What frustrations have you encountered?
+
+## Creative Testing (20 minutes)
+We'll now show you some design concepts and get your feedback.
+
+## Conclusion (10 minutes)
+To wrap up, I'd like to hear your final thoughts on what we've discussed today and any additional insights you'd like to share.
+    """
+  },
+  {
+    "id": "2",
+    "name": "Product Feature Feedback",
+    "status": "in-progress",
+    "participants": [
+      "1", "4", "5", "7"
+    ],
+    "date": "2023-06-15T10:00:00Z",
+    "duration": 90,
+    "topic": "product-feedback",
+    "discussionGuide": """
+# Discussion Guide: Product Feature Feedback
+
+## Introduction (5 minutes)
+Welcome to our focus group discussion. Today we'll be exploring your thoughts on our upcoming product features.
+
+## Warm-up Questions (10 minutes)
+Let's start by discussing your current experience with similar products.
+
+## Feature Exploration (30 minutes)
+We'll present several feature concepts and gather your feedback on each.
+
+## Prioritization Exercise (15 minutes)
+Help us understand which features would be most valuable to you.
+
+## Conclusion (10 minutes)
+Final thoughts and summary of the discussion.
+    """
+  },
+  {
+    "id": "3",
+    "name": "Marketing Campaign Testing",
+    "status": "in-progress",
+    "participants": [
+      "2", "4", "7"
+    ],
+    "date": "2023-06-12T15:30:00Z",
+    "duration": 45,
+    "topic": "creative-testing",
+    "discussionGuide": """
+# Discussion Guide: Marketing Campaign Testing
+
+## Introduction (5 minutes)
+Welcome everyone. Today we're evaluating some marketing campaign concepts.
+
+## Warm-up (5 minutes)
+Brief discussion about marketing campaigns that have caught your attention recently.
+
+## Campaign Concept Review (25 minutes)
+We'll review several campaign directions and gather your impressions.
+
+## Message Effectiveness (10 minutes)
+Discussion about which messages resonate most strongly and why.
+
+## Conclusion (5 minutes)
+Final impressions and recommendations.
+    """
+  }
+]
+
+def connect_to_mongodb():
+    """Connect to MongoDB with or without authentication"""
+    print("Connecting to MongoDB...")
+    
+    # Try with MongoDB default credentials first (widely used standard defaults)
+    standard_credentials = [
+        {"user": "admin", "pass": "admin", "db": "admin"},
+        {"user": "mongodb", "pass": "mongodb", "db": "admin"},
+        {"user": "root", "pass": "root", "db": "admin"},
+        {"user": "user", "pass": "pass", "db": "admin"}
+    ]
+    
+    # Try each set of standard credentials
+    for creds in standard_credentials:
+        try:
+            uri = f"mongodb://{creds['user']}:{creds['pass']}@localhost:27017/semblance_db?authSource={creds['db']}"
+            client = MongoClient(uri, serverSelectionTimeoutMS=2000)
+            db = client.semblance_db
+            # Test the connection with a simple command
+            db.command('ping')
+            print(f"Successfully connected to MongoDB with standard credentials ({creds['user']})")
+            return client, db
+        except Exception as e:
+            # Continue trying other credentials
+            pass
+    
+    print("Could not connect with standard credentials")
+    
+    # Try connecting without auth (in case auth is not required)
+    try:
+        client = MongoClient('mongodb://localhost:27017', serverSelectionTimeoutMS=2000)
+        db = client.semblance_db
+        # Try to perform an operation that requires auth to verify
+        # we actually have write access (ping might succeed without auth)
+        result = db.command('buildInfo')
+        if result:
+            # Try to create a test document to verify write access
+            test_result = db.test_collection.insert_one({"test": "auth"})
+            db.test_collection.delete_one({"_id": test_result.inserted_id})
+            print("Successfully connected to MongoDB without authentication")
+            return client, db
+    except Exception as e:
+        print(f"Could not connect without auth: {e}")
+    
+    # Try with environment variables
+    mongo_user = os.environ.get('MONGO_USER')
+    mongo_pass = os.environ.get('MONGO_PASS')
+    if mongo_user and mongo_pass:
+        try:
+            uri = f"mongodb://{mongo_user}:{mongo_pass}@localhost:27017/semblance_db?authSource=admin"
+            client = MongoClient(uri, serverSelectionTimeoutMS=2000)
+            db = client.semblance_db
+            # Test the connection with an operation
+            db.command('ping')
+            print(f"Successfully connected to MongoDB with environment credentials")
+            return client, db
+        except Exception as e:
+            print(f"Could not connect with environment credentials: {e}")
+    
+    # Ask for credentials interactively
+    print("\nMongoDB requires authentication.")
+    print("Please enter your MongoDB credentials:")
+    username = input("MongoDB username: ")
+    password = getpass("MongoDB password: ")
+    
+    try:
+        uri = f"mongodb://{username}:{password}@localhost:27017/semblance_db?authSource=admin"
+        client = MongoClient(uri, serverSelectionTimeoutMS=2000)
+        db = client.semblance_db
+        db.command('ping')
+        print("Successfully connected to MongoDB with provided credentials")
+        return client, db
+    except Exception as e:
+        print(f"Error connecting with provided credentials: {e}")
+        print("Could not connect to MongoDB. Please check your credentials.")
+        sys.exit(1)
+
+def create_default_user(db):
+    """Create a default admin user if it doesn't exist"""
+    try:
+        # Check if user exists
+        existing_user = db.users.find_one({"username": "admin"})
+        if existing_user:
+            print("Default admin user already exists")
+            return existing_user["_id"]
+        
+        # Create a simple password hash (in a real app, use proper password hashing)
+        # For this script, using a simple md5 hash (not secure for production!)
+        import hashlib
+        password_hash = hashlib.md5("admin".encode()).hexdigest()
+        
+        # Create user
+        user = {
+            "username": "admin",
+            "email": "admin@example.com",
+            "password": password_hash,
+            "created_at": datetime.datetime.now().isoformat()
+        }
+        result = db.users.insert_one(user)
+        print("Created default admin user")
+        return result.inserted_id
+    except Exception as e:
+        print(f"Error creating default user: {e}")
+        # Return a temporary ObjectId
+        return ObjectId()
+
+def main():
+    # Connect to MongoDB
+    client, db = connect_to_mongodb()
+    
+    # Create default user
+    user_id = create_default_user(db)
+    
+    # Clear existing data
+    try:
+        db.personas.delete_many({})
+        db.focus_groups.delete_many({})
+        print("Cleared existing personas and focus groups")
+    except Exception as e:
+        print(f"Warning: Could not clear collections: {e}")
+    
+    # Map from frontend IDs to MongoDB IDs
+    id_mapping = {}
+    
+    # Import personas
+    print("\nAdding personas...")
+    for persona_data in sample_personas:
+        try:
+            # Remove frontend ID
+            frontend_id = persona_data.pop("id")
+            
+            # Add metadata
+            persona_data["created_by"] = str(user_id)
+            persona_data["created_at"] = datetime.datetime.now()
+            
+            # Insert into MongoDB
+            result = db.personas.insert_one(persona_data)
+            mongo_id = result.inserted_id
+            
+            # Store mapping
+            id_mapping[frontend_id] = mongo_id
+            
+            print(f"Imported persona: {persona_data.get('name')} (frontend ID: {frontend_id} → MongoDB ID: {mongo_id})")
+        except Exception as e:
+            print(f"Error importing persona: {e}")
+    
+    # Import focus groups
+    print("\nAdding focus groups...")
+    for focus_group_data in sample_focus_groups:
+        try:
+            # Remove frontend ID
+            frontend_id = focus_group_data.pop("id")
+            
+            # Map participants
+            frontend_participants = focus_group_data.pop("participants", [])
+            participants = []
+            for p_id in frontend_participants:
+                if p_id in id_mapping:
+                    participants.append(id_mapping[p_id])
+            focus_group_data["participants"] = participants
+            
+            # Add metadata
+            focus_group_data["created_by"] = str(user_id)
+            focus_group_data["created_at"] = datetime.datetime.now()
+            
+            # Insert into MongoDB
+            result = db.focus_groups.insert_one(focus_group_data)
+            mongo_id = result.inserted_id
+            
+            print(f"Imported focus group: {focus_group_data.get('name')} (frontend ID: {frontend_id} → MongoDB ID: {mongo_id})")
+            print(f"  - Participants: {len(participants)}")
+        except Exception as e:
+            print(f"Error importing focus group: {e}")
+    
+    print("\nData import complete!")
+    client.close()
+
+if __name__ == "__main__":
+    main()

backend/scripts/setup_mongodb.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+
+# Define colors for readable output
+GREEN="\033[0;32m"
+RED="\033[0;31m"
+YELLOW="\033[0;33m"
+BLUE="\033[0;34m"
+NC="\033[0m" # No Color
+
+echo -e "${BLUE}===== MongoDB Setup Script =====${NC}"
+echo -e "This script will help set up MongoDB for development with the Semblance app"
+
+# Check if MongoDB is running
+if ! pgrep -x "mongod" > /dev/null; then
+    echo -e "${YELLOW}MongoDB is not running. Attempting to start MongoDB...${NC}"
+    if [[ "$OSTYPE" == "darwin"* ]]; then
+        # macOS
+        brew services start mongodb-community || mongod --config /usr/local/etc/mongod.conf --fork
+    else
+        # Linux
+        sudo systemctl start mongod || sudo service mongod start
+    fi
+    
+    # Wait for MongoDB to start
+    sleep 3
+    
+    # Check again
+    if ! pgrep -x "mongod" > /dev/null; then
+        echo -e "${RED}Failed to start MongoDB. Please start it manually before running this script.${NC}"
+        exit 1
+    else
+        echo -e "${GREEN}MongoDB started successfully.${NC}"
+    fi
+else
+    echo -e "${GREEN}MongoDB is already running.${NC}"
+fi
+
+echo -e "${YELLOW}Setting up MongoDB for development (no authentication)...${NC}"
+
+# Connect to MongoDB and disable authentication if enabled
+mongo admin --eval 'db.disableAuth = function() { db.getSiblingDB("admin").system.users.remove({}); db.getSiblingDB("admin").system.version.remove({ "_id": "authSchema" }); db.getSiblingDB("admin").system.version.insert({ "_id": "authSchema", "currentVersion": 3 }); print("Authentication has been disabled. Please restart MongoDB for changes to take effect."); }; try { db.disableAuth(); } catch (e) { print("Error disabling auth: " + e); }'
+
+echo -e "${YELLOW}Creating semblance_db database and collections...${NC}"
+
+# Create database and collections
+mongo --eval 'db = db.getSiblingDB("semblance_db"); db.createCollection("users"); db.createCollection("personas"); db.createCollection("focus_groups");'
+
+echo -e "${GREEN}MongoDB setup completed. The database is now ready for development.${NC}"
+echo -e "${YELLOW}Note: You may need to restart MongoDB for all changes to take effect:${NC}"
+echo -e "  - On macOS: brew services restart mongodb-community"
+echo -e "  - On Linux: sudo systemctl restart mongod"

backend/tests/__init__.py

@@ -0,0 +1,3 @@
+"""
+Test package for Synthetic Society backend
+"""

backend/tests/test_focus_group_ai.py

@@ -0,0 +1,201 @@
+"""
+Tests for the Focus Group AI API endpoints
+"""
+
+import unittest
+import json
+from unittest.mock import patch, MagicMock
+from flask import Flask
+from app import create_app
+from app.routes.focus_group_ai import focus_group_ai_bp
+from app.services.focus_group_response_service import FocusGroupResponseError
+
+class TestFocusGroupAIRoutes(unittest.TestCase):
+    """Test cases for the Focus Group AI API endpoints"""
+    
+    def setUp(self):
+        """Set up test Flask app and client"""
+        self.app = create_app()
+        self.app.config['TESTING'] = True
+        self.app.config['JWT_REQUIRED'] = False  # Disable JWT for testing
+        self.client = self.app.test_client()
+        
+    @patch('app.routes.focus_group_ai.FocusGroup.find_by_id')
+    @patch('app.routes.focus_group_ai.Persona.find_by_id')
+    @patch('app.routes.focus_group_ai.FocusGroup.get_messages')
+    @patch('app.routes.focus_group_ai.generate_persona_response')
+    @patch('app.routes.focus_group_ai.FocusGroup.add_message')
+    def test_generate_ai_response_success(self, mock_add_message, mock_generate_response, 
+                                         mock_get_messages, mock_find_persona, mock_find_focus_group):
+        """Test successful AI response generation"""
+        # Mock data
+        focus_group_id = '123456789'
+        persona_id = '987654321'
+        current_topic = 'What do you think about our new product?'
+        
+        # Mock focus group with the persona as a participant
+        mock_focus_group = {
+            '_id': focus_group_id,
+            'name': 'Test Focus Group',
+            'discussionGuide': 'This is a test discussion guide',
+            'participants': [persona_id]
+        }
+        mock_find_focus_group.return_value = mock_focus_group
+        
+        # Mock persona data
+        mock_persona = {
+            '_id': persona_id,
+            'name': 'Jane Doe',
+            'age': '30-35',
+            'gender': 'Female'
+        }
+        mock_find_persona.return_value = mock_persona
+        
+        # Mock previous messages
+        mock_get_messages.return_value = [
+            {'senderId': 'moderator', 'text': 'Welcome everyone!', 'type': 'system'},
+            {'senderId': 'moderator', 'text': 'Let\'s discuss our new product', 'type': 'question'}
+        ]
+        
+        # Mock response generation
+        mock_generate_response.return_value = "I think the new product looks promising. I like its features."
+        
+        # Mock message saving
+        mock_add_message.return_value = 'new_message_id'
+        
+        # Test request
+        response = self.client.post(
+            '/api/focus-group-ai/generate-response',
+            json={
+                'focus_group_id': focus_group_id,
+                'persona_id': persona_id,
+                'current_topic': current_topic
+            }
+        )
+        
+        # Assertions
+        self.assertEqual(response.status_code, 200)
+        data = json.loads(response.data)
+        self.assertIn('message', data)
+        self.assertIn('response', data)
+        self.assertEqual(data['response'], "I think the new product looks promising. I like its features.")
+        self.assertEqual(data['message_id'], 'new_message_id')
+        
+        # Verify function calls
+        mock_find_focus_group.assert_called_once_with(focus_group_id)
+        mock_find_persona.assert_called_once_with(persona_id)
+        mock_get_messages.assert_called_once_with(focus_group_id)
+        mock_generate_response.assert_called_once()
+        mock_add_message.assert_called_once()
+        
+    @patch('app.routes.focus_group_ai.FocusGroup.find_by_id')
+    def test_generate_ai_response_focus_group_not_found(self, mock_find_focus_group):
+        """Test response when focus group is not found"""
+        mock_find_focus_group.return_value = None
+        
+        response = self.client.post(
+            '/api/focus-group-ai/generate-response',
+            json={
+                'focus_group_id': 'nonexistent_id',
+                'persona_id': '987654321',
+                'current_topic': 'Test topic'
+            }
+        )
+        
+        self.assertEqual(response.status_code, 404)
+        data = json.loads(response.data)
+        self.assertIn('error', data)
+        self.assertEqual(data['error'], 'Focus group not found')
+        
+    @patch('app.routes.focus_group_ai.FocusGroup.find_by_id')
+    @patch('app.routes.focus_group_ai.Persona.find_by_id')
+    def test_generate_ai_response_persona_not_found(self, mock_find_persona, mock_find_focus_group):
+        """Test response when persona is not found"""
+        # Mock focus group
+        mock_find_focus_group.return_value = {'_id': '123', 'name': 'Test Focus Group'}
+        
+        # Mock persona not found
+        mock_find_persona.return_value = None
+        
+        response = self.client.post(
+            '/api/focus-group-ai/generate-response',
+            json={
+                'focus_group_id': '123',
+                'persona_id': 'nonexistent_id',
+                'current_topic': 'Test topic'
+            }
+        )
+        
+        self.assertEqual(response.status_code, 404)
+        data = json.loads(response.data)
+        self.assertIn('error', data)
+        self.assertEqual(data['error'], 'Persona not found')
+        
+    @patch('app.routes.focus_group_ai.FocusGroup.find_by_id')
+    @patch('app.routes.focus_group_ai.Persona.find_by_id')
+    def test_generate_ai_response_persona_not_in_focus_group(self, mock_find_persona, mock_find_focus_group):
+        """Test response when persona is not in the focus group"""
+        # Mock focus group with no participants
+        mock_find_focus_group.return_value = {
+            '_id': '123', 
+            'name': 'Test Focus Group',
+            'participants': ['different_persona_id']
+        }
+        
+        # Mock persona
+        mock_find_persona.return_value = {'_id': 'persona_id', 'name': 'Test Persona'}
+        
+        response = self.client.post(
+            '/api/focus-group-ai/generate-response',
+            json={
+                'focus_group_id': '123',
+                'persona_id': 'persona_id',
+                'current_topic': 'Test topic'
+            }
+        )
+        
+        self.assertEqual(response.status_code, 400)
+        data = json.loads(response.data)
+        self.assertIn('error', data)
+        self.assertEqual(data['error'], 'Persona is not a participant in this focus group')
+        
+    @patch('app.routes.focus_group_ai.FocusGroup.find_by_id')
+    @patch('app.routes.focus_group_ai.Persona.find_by_id')
+    @patch('app.routes.focus_group_ai.FocusGroup.get_messages')
+    @patch('app.routes.focus_group_ai.generate_persona_response')
+    def test_generate_ai_response_llm_error(self, mock_generate_response, mock_get_messages, 
+                                          mock_find_persona, mock_find_focus_group):
+        """Test handling of LLM service errors"""
+        # Mock focus group with the persona as a participant
+        mock_find_focus_group.return_value = {
+            '_id': '123',
+            'name': 'Test Focus Group',
+            'participants': ['persona_id']
+        }
+        
+        # Mock persona
+        mock_find_persona.return_value = {'_id': 'persona_id', 'name': 'Test Persona'}
+        
+        # Mock messages
+        mock_get_messages.return_value = []
+        
+        # Mock LLM error
+        mock_generate_response.side_effect = FocusGroupResponseError("LLM service unavailable")
+        
+        response = self.client.post(
+            '/api/focus-group-ai/generate-response',
+            json={
+                'focus_group_id': '123',
+                'persona_id': 'persona_id',
+                'current_topic': 'Test topic'
+            }
+        )
+        
+        self.assertEqual(response.status_code, 500)
+        data = json.loads(response.data)
+        self.assertIn('error', data)
+        self.assertEqual(data['error'], 'Failed to generate response')
+        self.assertIn('message', data)
+        
+if __name__ == '__main__':
+    unittest.main()

backend/tests/test_focus_group_response_service.py

@@ -0,0 +1,166 @@
+"""
+Tests for the Focus Group Response Service
+"""
+
+import unittest
+from unittest.mock import patch, MagicMock
+from app.services.focus_group_response_service import (
+    generate_persona_response,
+    FocusGroupResponseError,
+    _format_persona_details,
+    _format_previous_messages
+)
+from app.services.llm_service import LLMServiceError
+
+class TestFocusGroupResponseService(unittest.TestCase):
+    """Test cases for the Focus Group Response Service"""
+    
+    
+    def test_format_previous_messages(self):
+        """Test formatting previous messages"""
+        # No messages
+        self.assertEqual(_format_previous_messages([]), "No previous messages.")
+        
+        # With messages
+        messages = [
+            {"senderId": "moderator", "text": "Welcome everyone!", "type": "question"},
+            {"senderId": "system", "text": "New participant joined", "type": "system"},
+            {"senderId": "persona-123", "text": "Hello, I'm excited to be here", "type": "response"}
+        ]
+        
+        formatted = _format_previous_messages(messages)
+        self.assertIn("MODERATOR (moderator): Welcome everyone!", formatted)
+        self.assertIn("SYSTEM: New participant joined", formatted)
+        self.assertIn("persona-123: Hello, I'm excited to be here", formatted)
+        
+        # Many messages (should be limited to 50)
+        many_messages = [{"senderId": f"person-{i}", "text": f"Message {i}"} for i in range(60)]
+        formatted = _format_previous_messages(many_messages)
+        # Should only contain the 50 most recent messages
+        self.assertEqual(formatted.count("\n") + 1, 50)
+    
+    def test_format_persona_details(self):
+        """Test formatting persona details"""
+        # Basic persona
+        basic_persona = {
+            "name": "John Doe",
+            "age": "30-35",
+            "gender": "Male",
+            "occupation": "Software Engineer",
+            "education": "Bachelor's Degree",
+            "location": "San Francisco, USA",
+            "personality": "Analytical and introverted"
+        }
+        
+        formatted = _format_persona_details(basic_persona)
+        for key, value in basic_persona.items():
+            self.assertIn(f"{key.title()}: {value}", formatted)
+        
+        # Complex persona with OCEAN traits
+        complex_persona = {
+            **basic_persona,
+            "oceanTraits": {
+                "openness": 80,
+                "conscientiousness": 70,
+                "extraversion": 30,
+                "agreeableness": 60,
+                "neuroticism": 40
+            },
+            "goals": ["Learn new skills", "Get promoted"],
+            "frustrations": ["Bureaucracy", "Meetings"],
+            "motivations": ["Technical excellence", "Impact"],
+            "thinkFeelDo": {
+                "thinks": ["How can I optimize this?", "What's the underlying pattern?"],
+                "feels": ["Excited by technical challenges", "Frustrated by inefficiency"],
+                "does": ["Researches solutions thoroughly", "Creates detailed plans"]
+            }
+        }
+        
+        formatted = _format_persona_details(complex_persona)
+        
+        # Check OCEAN traits
+        self.assertIn("OCEAN Traits:", formatted)
+        self.assertIn("- Openness: 80/100", formatted)
+        
+        # Check goals
+        self.assertIn("Goals:", formatted)
+        for goal in complex_persona["goals"]:
+            self.assertIn(f"- {goal}", formatted)
+        
+        # Check think/feel/do
+        self.assertIn("Thinks:", formatted)
+        self.assertIn("Feels:", formatted)
+        self.assertIn("Does:", formatted)
+    
+    @patch('app.services.focus_group_response_service.LLMService.generate_content')
+    def test_generate_persona_response(self, mock_generate_content):
+        """Test generating a persona response"""
+        # Setup test data
+        persona = {
+            "name": "Jane Doe",
+            "age": "30-35",
+            "gender": "Female",
+            "occupation": "Product Manager",
+            "education": "Master's Degree",
+            "location": "Boston, USA",
+            "personality": "Confident and analytical"
+        }
+        
+        discussion_guide = "This focus group is about product features"
+        current_topic = "What do you think about the new dashboard?"
+        previous_messages = [
+            {"senderId": "moderator", "text": "Welcome everyone", "type": "system"},
+            {"senderId": "moderator", "text": "Let's discuss the dashboard", "type": "question"}
+        ]
+        
+        # Mock response
+        mock_generate_content.return_value = "I find the dashboard intuitive and well-organized."
+        
+        # Test function
+        response = generate_persona_response(
+            persona=persona,
+            current_topic=current_topic,
+            previous_messages=previous_messages,
+            temperature=0.5
+        )
+        
+        # Assertions
+        self.assertEqual(response, "I find the dashboard intuitive and well-organized.")
+        mock_generate_content.assert_called_once()
+        
+        # Check prompt content
+        call_args = mock_generate_content.call_args[1]
+        self.assertEqual(call_args["temperature"], 0.5)
+        prompt = call_args["prompt"]
+        
+        # Prompt should contain persona details
+        self.assertIn("Name: Jane Doe", prompt)
+        self.assertIn("Gender: Female", prompt)
+        
+        # Prompt should contain discussion guide
+        self.assertIn("This focus group is about product features", prompt)
+        
+        # Prompt should contain current topic
+        self.assertIn("What do you think about the new dashboard?", prompt)
+        
+        # Prompt should contain previous messages
+        self.assertIn("MODERATOR (moderator): Let's discuss the dashboard", prompt)
+    
+    @patch('app.services.focus_group_response_service.LLMService.generate_content')
+    def test_generate_persona_response_error(self, mock_generate_content):
+        """Test error handling in persona response generation"""
+        # Setup mock to raise an exception
+        mock_generate_content.side_effect = LLMServiceError("LLM service unavailable")
+        
+        # Test error propagation
+        with self.assertRaises(FocusGroupResponseError) as context:
+            generate_persona_response(
+                persona={"name": "Test Persona"},
+                current_topic="Test topic",
+                previous_messages=[]
+            )
+        
+        self.assertIn("Error generating persona response", str(context.exception))
+
+if __name__ == '__main__':
+    unittest.main()

backend/tests/test_llm_service.py

@@ -0,0 +1,145 @@
+"""
+Tests for the LLM Service
+"""
+
+import unittest
+from unittest.mock import patch, MagicMock
+import json
+from app.services.llm_service import LLMService, LLMServiceError
+
+class TestLLMService(unittest.TestCase):
+    """Test cases for the LLM Service"""
+    
+    @patch('app.services.llm_service.genai.GenerativeModel')
+    def test_get_model(self, mock_generative_model):
+        """Test getting a Gemini model"""
+        # Setup mock
+        mock_model = MagicMock()
+        mock_generative_model.return_value = mock_model
+        
+        # Test with default model
+        model = LLMService.get_model()
+        mock_generative_model.assert_called_once()
+        self.assertEqual(model, mock_model)
+        
+        # Reset mock
+        mock_generative_model.reset_mock()
+        
+        # Test with custom model
+        custom_model = "custom-model-name"
+        model = LLMService.get_model(custom_model)
+        mock_generative_model.assert_called_once_with(custom_model)
+        self.assertEqual(model, mock_model)
+    
+    @patch('app.services.llm_service.LLMService.get_model')
+    def test_generate_content(self, mock_get_model):
+        """Test generating content with the LLM"""
+        # Setup mock
+        mock_model = MagicMock()
+        mock_response = MagicMock()
+        mock_response.text = "Generated text response"
+        mock_model.generate_content.return_value = mock_response
+        mock_get_model.return_value = mock_model
+        
+        # Test with default parameters
+        prompt = "Test prompt"
+        response = LLMService.generate_content(prompt)
+        
+        mock_get_model.assert_called_once()
+        mock_model.generate_content.assert_called_once()
+        self.assertEqual(response, "Generated text response")
+        
+        # Test with custom parameters
+        mock_get_model.reset_mock()
+        mock_model.generate_content.reset_mock()
+        
+        response = LLMService.generate_content(
+            prompt="Custom prompt",
+            temperature=0.5,
+            max_tokens=100,
+            model_name="custom-model"
+        )
+        
+        mock_get_model.assert_called_once_with("custom-model")
+        mock_model.generate_content.assert_called_once()
+        self.assertEqual(response, "Generated text response")
+    
+    @patch('app.services.llm_service.LLMService.get_model')
+    def test_generate_content_error(self, mock_get_model):
+        """Test error handling in generate_content"""
+        # Setup mock to raise an exception
+        mock_model = MagicMock()
+        mock_model.generate_content.side_effect = Exception("Model error")
+        mock_get_model.return_value = mock_model
+        
+        # Test error handling
+        with self.assertRaises(LLMServiceError) as context:
+            LLMService.generate_content("Test prompt")
+        
+        self.assertIn("Error generating content", str(context.exception))
+    
+    def test_parse_json_response_valid(self):
+        """Test parsing valid JSON responses"""
+        # Test with clean JSON
+        clean_json = '{"key": "value", "number": 42}'
+        result = LLMService.parse_json_response(clean_json)
+        expected = {"key": "value", "number": 42}
+        self.assertEqual(result, expected)
+        
+        # Test with JSON in markdown code block
+        markdown_json = '```json\n{"key": "value", "number": 42}\n```'
+        result = LLMService.parse_json_response(markdown_json)
+        self.assertEqual(result, expected)
+        
+        # Test with JSON in generic code block
+        generic_code_block = '```\n{"key": "value", "number": 42}\n```'
+        result = LLMService.parse_json_response(generic_code_block)
+        self.assertEqual(result, expected)
+    
+    def test_parse_json_response_invalid(self):
+        """Test parsing invalid JSON responses"""
+        invalid_json = 'This is not JSON'
+        
+        with self.assertRaises(LLMServiceError) as context:
+            LLMService.parse_json_response(invalid_json)
+        
+        self.assertIn("Failed to parse JSON response", str(context.exception))
+    
+    @patch('app.services.llm_service.LLMService.generate_content')
+    @patch('app.services.llm_service.LLMService.parse_json_response')
+    def test_generate_structured_response(self, mock_parse_json, mock_generate_content):
+        """Test generating a structured JSON response"""
+        # Setup mocks
+        mock_generate_content.return_value = '{"result": "success"}'
+        mock_parse_json.return_value = {"result": "success"}
+        
+        # Test
+        result = LLMService.generate_structured_response(
+            prompt="Generate JSON",
+            temperature=0.5
+        )
+        
+        mock_generate_content.assert_called_once_with(
+            prompt="Generate JSON",
+            temperature=0.5,
+            max_tokens=None,
+            model_name=None,
+            system_prompt=None
+        )
+        mock_parse_json.assert_called_once_with('{"result": "success"}')
+        self.assertEqual(result, {"result": "success"})
+    
+    @patch('app.services.llm_service.LLMService.generate_content')
+    def test_generate_structured_response_error(self, mock_generate_content):
+        """Test error handling in generate_structured_response"""
+        # Setup mock to raise an exception
+        mock_generate_content.side_effect = LLMServiceError("Generation failed")
+        
+        # Test error propagation
+        with self.assertRaises(LLMServiceError) as context:
+            LLMService.generate_structured_response("Generate JSON")
+        
+        self.assertEqual(str(context.exception), "Generation failed")
+
+if __name__ == '__main__':
+    unittest.main()

components.json

@@ -0,0 +1,20 @@
+{
+  "$schema": "https://ui.shadcn.com/schema.json",
+  "style": "default",
+  "rsc": false,
+  "tsx": true,
+  "tailwind": {
+    "config": "tailwind.config.ts",
+    "css": "src/index.css",
+    "baseColor": "slate",
+    "cssVariables": true,
+    "prefix": ""
+  },
+  "aliases": {
+    "components": "@/components",
+    "utils": "@/lib/utils",
+    "ui": "@/components/ui",
+    "lib": "@/lib",
+    "hooks": "@/hooks"
+  }
+}

dist/.htaccess

@@ -0,0 +1,9 @@
+# Apache .htaccess file for SPA routing
+RewriteEngine On
+
+# Handle client-side routing for /semblance/
+RewriteBase /semblance/
+RewriteRule ^index\.html$ - [L]
+RewriteCond %{REQUEST_FILENAME} !-f
+RewriteCond %{REQUEST_FILENAME} !-d
+RewriteRule . /semblance/index.html [L]