ideas-generator/README.md

# Barclays Ideas Generator - Current Architecture & 2025 Transition Plan

## Current Application Overview

The **Barclays Ideas Generator** is an enterprise-grade conversational AI platform that provides employees with access to multiple AI assistants for ideation and business support. The system uses a sophisticated cloud-based architecture with proper authentication and security measures.

### Key Features

- **Multiple AI Assistants**: Users can select from different pre-configured AI assistants with unique personalities and capabilities
- **Conversation Management**: Persistent chat threads with history, titles, and deletion capabilities
- **Tone of Voice Control**: Selectable tone-of-voice options to customize AI responses
- **Security-First Design**: Bank detail masking, cybersecurity term filtering, and enterprise authentication
- **Responsive Interface**: Clean, modern chat interface with sidebar navigation

### Current Architecture

```
Frontend (Web App) → GCF Proxy → Make.com Webhook → AI Service → Response Chain
                   ↓
            Microsoft Azure AD (Authentication)
```

#### Frontend Components

**Core Files:**
- `index.html` - Main application shell with Microsoft MSAL authentication
- `js/script.js` - Core application logic and AI interaction
- `js/variables.js` - Configuration variables (Make.com webhook URL)
- `js/html.js` - HTML templates for dynamic content
- `style.css` - Application styling

**Key JavaScript Functions:**
- `sendMessage()` - Handles user input and AI communication
- `getAssistants()` - Retrieves available AI assistants
- `getConversations()` - Loads conversation history
- `maskUKBankDetails()` - Security function for data sanitization

#### Backend Services

**Google Cloud Function Proxy** (`GCF/index.js`):
- CORS handling for cross-origin requests
- Authentication token management
- Request forwarding to Make.com webhook

**Make.com Integration** (`js/variables.js`):
- Primary AI processing endpoint: `https://hook.us1.make.celonis.com/htn0fepeoai19d1unx6fqm5qd5ptk5px`
- Handles conversation management, assistant selection, and AI responses

**API Endpoints:**
- `?GetConversations=True` - Retrieve user conversations
- `?GetAssistants=True` - Get available AI assistants
- `?GetMessages=True&ConversationID={id}` - Load conversation history
- `?DeleteConversation=True&ConversationID={id}` - Remove conversations
- Main chat endpoint with parameters: `ConversationID`, `AssistantKey`, `TOV_Key`, `Message`

#### Authentication & Security

**Microsoft Azure AD Integration:**
- Client ID: `9079054c-9620-4757-a256-23413042f1ef`
- Tenant ID: `e519c2e6-bc6d-4fdf-8d9c-923c2f002385`
- Redirect URI: `https://ai-sandbox.oliver.solutions/ideas-sparkplug/index.html`

**Security Features:**
- **Data Sanitization**: Automatic masking of UK banking details (sort codes, account numbers, card numbers)
- **Content Filtering**: Cybersecurity term detection and masking
- **Session Management**: Secure cookie handling with HttpOnly, Secure, and SameSite flags
- **CORS Protection**: Configured allowed origins
- **Authentication Required**: All features require Microsoft AD login

### Current Limitations

1. **Indirect AI Integration**: Uses Make.com webhook instead of direct OpenAI API calls
2. **Complex Architecture**: Multiple proxy layers add latency and complexity
3. **Limited Control**: Cannot easily customize AI behavior or access advanced features
4. **Dependency on External Services**: Reliant on Make.com and Google Cloud Functions
5. **Authentication Overhead**: Enterprise authentication may be excessive for development

---

## 2025 Transition Plan: From OpenAI Assistants to Direct API Integration

### Transition Overview

**Goal**: Migrate from the current Make.com webhook architecture to direct OpenAI API completions for better control, performance, and feature access.

### Phase 1: Local Development Setup (Week 1)

#### 1.1 Authentication Bypass for Development
**Files to modify:**
- `index.html` (lines 11-32, 68-127) - Comment out MSAL authentication
- Create development flag to bypass login requirements
- Replace `thisUser` with a default development user

#### 1.2 Local Backend Creation
**New files to create:**
- `server/` directory with Node.js/Express backend
- `server/index.js` - Main server file with OpenAI integration
- `server/config.js` - Configuration management
- `server/routes/` - API route handlers
- `server/package.json` - Dependencies

**Key dependencies:**
```json
{
  "express": "^4.18.2",
  "cors": "^2.8.5",
  "openai": "^4.0.0",
  "dotenv": "^16.3.1",
  "express-rate-limit": "^7.1.5"
}
```

#### 1.3 Environment Configuration
**New files:**
- `.env` - OpenAI API key and configuration
- `.env.example` - Template for environment variables

### Phase 2: API Endpoint Migration (Week 2)

#### 2.1 Replace Make.com Webhook
**Modify `js/variables.js`:**
```javascript
// OLD:
const make_url = "https://hook.us1.make.celonis.com/htn0fepeoai19d1unx6fqm5qd5ptk5px";

// NEW:
const make_url = "http://localhost:3000/api";
```

#### 2.2 Create Local API Endpoints

**Endpoint mapping:**
| Current Make.com Parameter | New Local Endpoint | Method |
|---------------------------|-------------------|--------|
| `?GetConversations=True` | `/api/conversations` | GET |
| `?GetAssistants=True` | `/api/assistants` | GET |
| `?GetMessages=True&ConversationID={id}` | `/api/conversations/{id}/messages` | GET |
| `?DeleteConversation=True&ConversationID={id}` | `/api/conversations/{id}` | DELETE |
| Main chat endpoint | `/api/chat` | POST |

#### 2.3 OpenAI Integration Strategy

**From OpenAI Assistants to Chat Completions:**

**Current (via Make.com):**
```
Assistant ID → Make.com → OpenAI Assistants API
```

**New (direct):**
```javascript
// Direct OpenAI Chat Completions
const completion = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: assistantSystemPrompt },
    ...conversationHistory,
    { role: "user", content: userMessage }
  ],
  temperature: 0.7,
  max_tokens: 1000
});
```

### Phase 3: Data Storage Implementation (Week 3)

#### 3.1 Local Database Setup
**Options:**
1. **SQLite** (recommended for development) - File-based, zero-config
2. **JSON files** - Simple file storage for prototyping
3. **PostgreSQL** - Full database for production-ready features

#### 3.2 Data Models
**Conversations:**
```javascript
{
  id: string,
  title: string,
  assistant_key: string,
  tov_key: string,
  user_id: string,
  created_at: timestamp,
  updated_at: timestamp
}
```

**Messages:**
```javascript
{
  id: string,
  conversation_id: string,
  role: 'user' | 'assistant',
  content: string,
  created_at: timestamp
}
```

**Assistants:**
```javascript
{
  key: string,
  name: string,
  system_prompt: string,
  initial_message: string,
  model: string,
  temperature: number
}
```

### Phase 4: Feature Enhancement (Week 4)

#### 4.1 Assistant Configuration
Replace hardcoded assistant selection with configurable system prompts:

**Current approach:**
- Assistant selection via `assistant_key`
- Limited customization through Make.com

**New approach:**
```javascript
const assistants = [
  {
    key: "creative_ideation",
    name: "Creative Ideation Assistant",
    system_prompt: "You are a creative business ideation assistant...",
    model: "gpt-4o",
    temperature: 0.8
  },
  {
    key: "analytical_advisor",
    name: "Analytical Business Advisor", 
    system_prompt: "You are a data-driven business analyst...",
    model: "gpt-4o",
    temperature: 0.3
  }
];
```

#### 4.2 Enhanced Features
- **Streaming Responses**: Real-time message streaming
- **Custom System Prompts**: Easily editable assistant personalities
- **Conversation Export**: Export chat history as PDF/JSON
- **Advanced Filtering**: Better content filtering and compliance
- **Usage Analytics**: Track API usage and costs

### Phase 5: Security & Compliance (Week 5)

#### 5.1 Data Protection
- Maintain existing `maskUKBankDetails()` function
- Add configurable content filtering
- Implement request/response logging for compliance

#### 5.2 Rate Limiting & Safety
```javascript
// Rate limiting
const rateLimit = require('express-rate-limit');
const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100 // limit each user to 100 requests per windowMs
});

// Content safety
const moderationResult = await openai.moderations.create({
  input: userMessage
});
```

### Implementation Checklist

#### Week 1: Foundation
- [ ] Create new branch `ideas-gen-2025`
- [ ] Set up local Node.js backend structure
- [ ] Install required dependencies
- [ ] Create basic Express server
- [ ] Comment out authentication for development
- [ ] Test basic frontend-backend communication

#### Week 2: API Migration
- [ ] Create conversation management endpoints
- [ ] Implement assistant selection API
- [ ] Set up OpenAI API integration
- [ ] Replace Make.com webhook calls
- [ ] Test message sending/receiving
- [ ] Implement basic error handling

#### Week 3: Data Persistence
- [ ] Choose and set up database solution
- [ ] Create data models and schemas
- [ ] Implement conversation storage
- [ ] Add message history functionality
- [ ] Test data persistence across sessions

#### Week 4: Feature Enhancement
- [ ] Implement configurable assistants
- [ ] Add tone-of-voice customization
- [ ] Create conversation export features
- [ ] Add streaming message responses
- [ ] Implement usage tracking

#### Week 5: Security & Polish
- [ ] Add comprehensive content filtering
- [ ] Implement rate limiting
- [ ] Add request logging
- [ ] Create admin panel for assistant management
- [ ] Performance optimization
- [ ] Documentation updates

### Questions for Clarification

1. **OpenAI Model Preferences**: Which OpenAI models would you prefer? (GPT-4o, GPT-4o-mini, etc.)
2. **Database Choice**: SQLite for simplicity or PostgreSQL for features?
3. **Assistant Configuration**: How many pre-configured assistants do you want?
4. **Streaming**: Should we implement real-time streaming responses?
5. **Authentication Timeline**: When do you want to re-enable authentication?
6. **Deployment**: Local development only or eventual cloud deployment?
7. **Legacy Support**: Should we maintain backward compatibility with existing conversations?

### Benefits of the New Architecture

1. **Direct Control**: Full control over AI behavior and responses
2. **Better Performance**: Eliminate proxy layers and reduce latency
3. **Cost Optimization**: Direct API usage for better cost management
4. **Enhanced Features**: Access to latest OpenAI features and models
5. **Easier Debugging**: Local development and debugging capabilities
6. **Customization**: Easy assistant personality and behavior modification
7. **Data Ownership**: Complete control over conversation data

This transition plan provides a clear path from the current Make.com-based architecture to a modern, direct OpenAI integration while maintaining the existing user experience and security standards.