ideas-generator/BACKEND_ARCHITECTURE.md

# Complete Backend Architecture - Make.com Workflow Analysis

## Overview

The **Creative Sidekick** backend is implemented as a sophisticated Make.com workflow that provides a complete conversational AI platform using OpenAI's Assistants API. This document provides a comprehensive technical breakdown of the current implementation.

## Workflow Architecture

```
Webhook Entry Point
        ↓
Authentication Check
        ↓
Request Router (GetMessages/GetConversations/GetAssistants/Chat)
        ↓
Data Operations (Search/Create/Update/Delete)
        ↓
OpenAI API Calls (Moderation/Threads/Assistants/Completions)
        ↓
Response Processing (Markdown/JSON/Aggregation)
        ↓
Return Formatted Response
```

## Data Model

### Datastore 1607: Assistants
```javascript
{
  "Assistant ID": "asst_xxx", // OpenAI Assistant ID
  "Name": "Creative Ideation Assistant",
  "Instructions": "System prompt for the assistant",
  "Model": "gpt-4-turbo", 
  "Deleted": false, // Soft delete flag
  "Initial Message": "Hello! I'm here to help with creative ideation."
}
```

### Datastore 1608: Conversations
```javascript
{
  "User_ID": "user@example.com",
  "Title": "Marketing Ideas", // Auto-generated by GPT-4
  "StartTime": "2024-01-01T10:00:00Z",
  "EndTime": "2024-01-01T11:00:00Z", 
  "Thread_ID": "thread_xxx", // OpenAI Thread ID
  "Assistant_ID": "asst_xxx",
  "Assistant_Key": "creative_ideation",
  "Conversation_ID": "conv_uuid",
  "Cost": 0.05, // Usage tracking
  "Assistant Setting": {...},
  "Brand Voice Setting": "standard"
}
```

### Datastore 1609: Messages
```javascript
{
  "Conversation_ID": "conv_uuid",
  "Role": "user" | "assistant",
  "Content": "<p>Formatted HTML content</p>", // Processed markdown
  "Content_NoFormatting": "Plain text content", // Fallback
  "TimeStamp": "2024-01-01T10:30:00Z"
}
```

## API Endpoints & Routing

### 1. GET /api/assistants (`?GetAssistants=True`)

**Workflow Path**: Module 175 → Module 73 (filtered: GetAssistants)
**Data Source**: Datastore 1607
**Logic**:
1. Validate authenticated user
2. Query assistants where `Deleted != true`
3. Format response as JSON array

**Response Format**:
```javascript
{
  "assistants": [
    {
      "key": "creative_ideation",
      "id": "asst_xxx", 
      "name": "Creative Ideation Assistant",
      "initial_message": "Hello! I'm here to help..."
    }
  ]
}
```

### 2. GET /api/conversations (`?GetConversations=True`)

**Workflow Path**: Module 175 → Module 73 (filtered: GetConversations)
**Data Source**: Datastore 1608
**Logic**:
1. Validate authenticated user
2. Query conversations for user, sorted by EndTime DESC
3. Aggregate into JSON array

**Response Format**:
```javascript
{
  "conversations": [
    {
      "id": "conv_uuid",
      "title": "Marketing Ideas",
      "assistant_key": "creative_ideation",
      "tov_key": "standard"
    }
  ]
}
```

### 3. GET /api/conversations/{id}/messages (`?GetMessages=True&ConversationID={id}`)

**Workflow Path**: Module 175 → Module 73 (filtered: GetMessages)
**Data Source**: Datastore 1609
**Logic**:
1. Validate authenticated user
2. Query messages for conversation, sorted by TimeStamp ASC
3. Aggregate messages into JSON array

**Response Format**:
```javascript
{
  "conversation_id": "conv_uuid",
  "messages": [
    {
      "role": "user",
      "content": "Help me with marketing ideas"
    },
    {
      "role": "assistant", 
      "content": "<p>Here are some creative marketing strategies...</p>"
    }
  ]
}
```

### 4. DELETE /api/conversations/{id} (`?DeleteConversation=True&ConversationID={id}`)

**Workflow Path**: Module 175 → Module 824 (Delete router)
**Logic**:
1. Validate authenticated user and conversation ownership
2. Soft delete conversation record
3. Return success confirmation

### 5. POST /api/chat (New Message)

**Parameters**:
- `ConversationID` (optional - empty for new conversation)
- `AssistantKey` (required)
- `TOV_Key` (required - tone of voice)
- `Message` (required - user message)

**Workflow Paths**:
- **New Conversation**: Module 1031 → Complex flow for thread creation
- **Existing Conversation**: Module 1048 → Simpler message append flow

## OpenAI Integration Details

### 1. Content Moderation (Module 161)
```javascript
// POST https://api.openai.com/v1/moderations
{
  "input": "User message content"
}

// Headers:
{
  "Authorization": "Bearer sk-ideas-sidekick-[API_KEY]",
  "OpenAI-Organization": "org-HSioKMud1tZBdpWhBjJE6SLe"
}
```

### 2. Thread Creation (Module 203)
```javascript
// POST https://api.openai.com/v1/threads
{
  "messages": [{
    "role": "user",
    "content": "Please use this tone of voice for your responses: [TOV_CONTENT]"
  }]
}

// Headers:
{
  "Authorization": "Bearer [API_KEY]",
  "OpenAI-Beta": "assistants=v1"
}
```

### 3. Assistant Message Processing (Modules 519/520)
```javascript
// Make.com OpenAI Connector: messageAssistantAdvanced
{
  "assistantId": "asst_xxx",
  "threadId": "thread_xxx", 
  "role": "user",
  "message": "User input content"
}
```

### 4. Title Generation (Module 497)
```javascript
// POST https://api.openai.com/v1/chat/completions
{
  "model": "gpt-4-turbo",
  "messages": [
    {
      "role": "system",
      "content": "You are a conversation title generator with decades of experience. It is extremely important that you only ever output a short single title on it's own."
    },
    {
      "role": "user", 
      "content": "I will provide you text of a conversation between two individuals named USER and ASSISTANT which you will use to generate an appropriate title. Do you understand?"
    },
    {
      "role": "assistant",
      "content": "Yes, I understand."
    },
    {
      "role": "user",
      "content": "In your next message, please respond only with a short title that is shorter than 4 words relating to this conversation. Reword titles to be shorter and more concise if needed. Never use quotation marks around the title and never use before text such as Title: or Conversation Title:. CHAT: USER: [USER_MESSAGE]."
    }
  ]
}
```

## Business Logic Flows

### New Conversation Flow
```
1. Authenticate user
2. Validate AssistantKey exists
3. Content moderation check (OpenAI Moderations API)
4. Retrieve assistant details from datastore 1607
5. Create OpenAI thread with TOV message (Threads API)
6. Create conversation record in datastore 1608 
7. Store user message in datastore 1609
8. Send message to OpenAI assistant (Assistants API)
9. Process response through markdown compiler
10. Store assistant response in datastore 1609
11. Generate title using GPT-4 (Chat Completions API)
12. Update conversation with title and thread_id
13. Return formatted response with conversation_id
```

### Existing Conversation Flow
```
1. Authenticate user  
2. Validate conversation ownership
3. Content moderation check
4. Store user message in datastore 1609
5. Send message to existing thread (Assistants API)
6. Process response through markdown compiler
7. Store assistant response in datastore 1609
8. Update conversation EndTime
9. Return formatted response
```

### Data Retrieval Flows
```
GetConversations:
1. Authenticate user
2. Query datastore 1608 filtered by user_id
3. Sort by EndTime DESC (most recent first)
4. Return JSON array

GetMessages:  
1. Authenticate user
2. Validate conversation ownership
3. Query datastore 1609 filtered by conversation_id
4. Sort by TimeStamp ASC (chronological order)
5. Return JSON array with conversation metadata

GetAssistants:
1. Authenticate user  
2. Query datastore 1607 where Deleted != true
3. Return JSON array with key, name, initial_message
```

## Security & Authentication

### Authentication Method
- Simple parameter-based authentication using `authenticateduser` field
- No JWT tokens or complex session management
- All datastore queries filtered by authenticated user ID

### Content Security
- **OpenAI Moderation**: All user messages checked for policy violations
- **User Isolation**: Strict user-based data filtering in all operations
- **Soft Deletes**: Assistants use deletion flags instead of hard deletes

### Error Handling
```javascript
// Authentication errors
{"error": "Unauthorized"}

// Missing assistant
{"error": "Error: Assistant Not Set"}

// Invalid conversation access
{"error": "Conversation not found"}
```

## Response Processing

### Markdown Compilation
- Assistant responses processed through markdown compiler
- HTML output stored in `Content` field
- Plain text backup stored in `Content_NoFormatting`

### JSON Formatting
- Consistent response structures across all endpoints
- Proper error handling with standardized error messages
- Aggregated list responses for conversations/messages/assistants

## Performance Considerations

### Data Access Patterns
- Conversations sorted by EndTime for recency
- Messages sorted by TimeStamp for chronological display
- Assistants filtered for active-only display

### Cost Tracking
- Usage costs tracked per conversation
- OpenAI API calls monitored and logged
- Separate title generation from main conversation flow

## Configuration

### OpenAI Settings
- **Organization ID**: `org-HSioKMud1tZBdpWhBjJE6SLe`
- **Primary Model**: GPT-4 Turbo for assistants
- **Title Generation**: GPT-4 Turbo with specific prompt engineering
- **API Version**: Assistants v1

### Tone of Voice Options
- **Standard**: Default tone
- **Pep**: Energetic/enthusiastic tone
- Configurable per conversation

### Assistant Configuration
- Multiple pre-configured assistants
- Custom system instructions per assistant
- Configurable initial messages
- Model selection per assistant (GPT-4 variants)

This architecture provides a complete, production-ready conversational AI platform with proper data persistence, security, and integration with OpenAI's most advanced capabilities.