Commit graph

105 commits

Author SHA1 Message Date
DJP
746a13f771 Add podcast queue - process one at a time to prevent timeouts 2025-10-02 16:23:26 -04:00
DJP
7aa30b57ac Increase Claude max_tokens to 8192 and timeout to 10min for podcast generation 2025-10-02 16:21:24 -04:00
DJP
9e80d77b2e Increase Claude timeout to 5 minutes for podcast generation 2025-10-02 15:21:09 -04:00
DJP
742784780b Fix Claude truncation with max_tokens=4096, improve podcast conversational tone 2025-10-02 14:07:16 -04:00
DJP
3268cfd02e Add JSON validation and better prompting for Claude/Gemini podcasts 2025-10-02 13:58:19 -04:00
DJP
6978fc07a7 Workaround for Claude/Gemini structured output - use JSON mode instead 2025-10-02 13:45:12 -04:00
DJP
ad09feef88 Fix synthesis parsing for all models - handle object vs JSON 2025-10-02 13:43:01 -04:00
DJP
10e23d70bd Fix podcast outline parsing for Claude/Gemini - handle object vs JSON string 2025-10-02 13:42:40 -04:00
DJP
ff93c1c7e2 Simplify readiness check - skip doc count, just test query response 2025-10-02 13:37:21 -04:00
DJP
ffbd7e2924 Fix: Use test query ONCE for readiness, detects when docs actually searchable 2025-10-02 13:32:05 -04:00
DJP
32d0548475 Clean up model dropdown text - remove status labels 2025-10-02 13:30:01 -04:00
DJP
f63e91a3c8 Replace test queries with direct pipeline status check - no more wasted API calls 2025-10-02 13:11:26 -04:00
DJP
bca37e5759 Add 10-minute timeout for indexing failures, progress bar, and long-wait warnings 2025-10-02 13:06:18 -04:00
DJP
d40f85ab8f Add comprehensive failure handling for document processing
Failure Detection:
- Better error handling in background_tasks.py
- Detailed error messages (300 char limit)
- Stack traces printed to logs
- Specific error types: Parse failed, Extraction failed, Processing error

UI Improvements:
- Failed documents shown in red error box
- Expandable list of failed documents with error details
- Shows exact error message from processing
- Auto-expanded to draw attention

User Actions:
- 🔄 Retry button (prompts to re-upload)
- 🗑️ Remove button (clears failed task from queue)
- Clear error messages explain what went wrong

Error Messages:
- "Parse failed: No content extracted" (empty PDF)
- "Extraction failed: No data returned" (LlamaExtract issue)
- "Processing error: [details]" (general errors)
- Truncated to 300 chars for readability

Benefits:
- No more infinite loops on failures
- Users know exactly what failed and why
- Can remove failed tasks to clean up UI
- Can retry by re-uploading
- Admin can see failures in dashboard

Testing:
- Upload corrupted file → Shows failure
- Upload unsupported format → Shows error
- Network error → Captured and displayed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 13:02:55 -04:00
DJP
3475d43151 Add failed document detection, error display, and retry/remove options 2025-10-02 13:02:22 -04:00
DJP
7a02674767 Remove min_length/max_length constraints for Gemini compatibility 2025-10-02 12:50:35 -04:00
DJP
811c5b4fc3 Fix: Only test pipeline readiness once, cache result, don't retest on every chat 2025-10-02 12:24:59 -04:00
DJP
c1c81c8d55 Add real pipeline readiness check - tests actual query response 2025-10-02 12:12:54 -04:00
DJP
be09801ecb Complete README overhaul with all AI models, features, and updated instructions 2025-10-02 11:58:23 -04:00
DJP
c24508860a Add file and extraction cleanup - complete LlamaCloud resource deletion 2025-10-02 11:40:38 -04:00
DJP
1fc94dc7d4 Fix cleanup script import path 2025-10-02 11:30:13 -04:00
DJP
e0711a6593 Add LlamaCloud pipeline cleanup on delete + one-time cleanup script
Pipeline Cleanup on Delete:
- Notebook deletion now cleans up LlamaCloud resources
- Deletes pipeline from LlamaCloud first
- Then deletes from database
- Prevents orphaned pipelines accumulating
- Saves storage costs and quota limits
- Shows progress: "Cleaning up LlamaCloud resources..."
- Graceful handling if pipeline already deleted

One-Time Cleanup Script:
- Created cleanup_llamacloud.py
- Deletes ALL pipelines from LlamaCloud account
- Safety prompt: Must type "DELETE ALL" to proceed
- Shows count of found vs deleted pipelines
- Lists all deleted pipeline IDs
- Use after deleting test notebooks

Usage:
python cleanup_llamacloud.py

Warning System:
- Clear warnings about what will be deleted
- Confirmation required
- Shows exactly what was cleaned up
- Error handling for safety

Benefits:
- Prevent orphaned resources in LlamaCloud
- Clean up quota limits
- Save costs
- Keep cloud account tidy
- Easy maintenance

Files Modified:
- pages/1_My_Notebooks.py (delete function)
- pipeline_manager.py (delete_pipeline, cleanup_all_orphaned_pipelines)
- cleanup_llamacloud.py (new script)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 11:25:39 -04:00
DJP
4edf62c4e3 Add Gemini 2.5 Pro option to UI dropdown 2025-10-02 11:21:23 -04:00
DJP
4fef829034 Update Gemini 2.5 Pro to use model name: gemini-2.5-pro 2025-10-02 11:18:40 -04:00
DJP
b093d03d69 Add Gemini 2.5 Pro back with correct model name (gemini-2.5-pro) 2025-10-02 11:18:14 -04:00
DJP
ac56369334 GPT-5 confirmed working! Make it default option 2025-10-02 11:14:58 -04:00
DJP
51d2aeb86b Add GPT-5 back as experimental option 2025-10-02 11:11:45 -04:00
DJP
e79c7e6ac9 Remove non-working models (GPT-5, Gemini 2.5), keep Claude 4.5, GPT-4o, Gemini 2.0, GPT-4 2025-10-02 11:10:47 -04:00
DJP
a60f3e96b1 Add model attribution to responses and better indexing wait messages 2025-10-02 11:04:51 -04:00
DJP
a3fba01f44 Add pipeline readiness check - blocks chat until documents are indexed 2025-10-02 10:57:39 -04:00
DJP
ddc52b9922 Add extensive debug logging to pipeline query response 2025-10-02 10:51:14 -04:00
DJP
af0508682c Add experimental GPT-5, Claude 4.5, Gemini 2.5 options alongside stable models 2025-10-02 10:39:01 -04:00
DJP
63e052cc8c Fix to actually available models: gpt-4o, claude-sonnet-4-20250514, gemini-2.0-flash 2025-10-02 10:23:36 -04:00
DJP
34d0781bd7 Fix to actual model names: gpt-5, claude-sonnet-4-5-20250929, gemini-pro-2.5 2025-10-02 10:17:43 -04:00
DJP
4d44e3ff6d Fix model names: gpt-4.1, claude-sonnet-4-0, gemini-2.0-flash 2025-10-02 10:09:11 -04:00
DJP
4e09d3a320 Add GPT-5 and Claude Sonnet 4.5 support - 4 AI models total 2025-10-02 09:56:06 -04:00
DJP
3932b5b139 Update to Gemini 2.5 Pro and add model indicator in chat 2025-10-02 09:49:22 -04:00
DJP
992640f8ea Add TRANSFORMATION.md - complete feature comparison and stats 2025-10-02 09:37:18 -04:00
DJP
89c6e6fca2 Add Gemini 2.0 Flash support with per-notebook model selection
Gemini Integration:
- Added Google Gemini 2.0 Flash as alternative to OpenAI GPT-4
- Per-notebook model selection (choose when creating)
- Model type stored in notebooks.model_type column
- Complete support for chat, synthesis, and podcasts

LLM Factory:
- Created llm_factory.py module
- get_llm_by_type('openai' | 'gemini')
- get_structured_llm() for pydantic outputs
- Model display names and emojis
- Cost information

Model Selection UI:
- Dropdown in notebook creation form
- Options: OpenAI GPT-4 or Gemini 2.0 Flash
- Shows cost comparison
- Model badge in notebook detail header
- Displays: "🤖 Using OpenAI GPT-4" or " Using Gemini 2.0 Flash"

Integration Points:
- pipeline_manager.py: Query engine uses selected model
- notebook_synthesis.py: Synthesis uses selected model
- background_tasks.py: Podcasts use selected model
- pages/3_Notebook_Chat.py: Chat uses selected model
- All functions pass model_type parameter

Cost Savings:
- Gemini 2.0 Flash: FREE tier available
- OpenAI GPT-4: ~$0.03/1K tokens
- Users can choose based on budget/quality needs

Dependencies Added:
- llama-index-llms-gemini>=0.3.0
- google-generativeai>=0.8.0

Note: Requires GOOGLE_API_KEY in .env (optional)
- If not set, Gemini notebooks will error
- OpenAI notebooks continue to work

Database:
- Added model_type VARCHAR(50) DEFAULT 'openai'
- Indexed for performance
- Existing notebooks default to 'openai'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 09:24:20 -04:00
DJP
be1671498e Add Word and PowerPoint document support (.docx, .pptx) 2025-10-02 09:12:40 -04:00
DJP
33adc2504a Final polish and completion of Sandbox-NotebookLM
Summary of Complete Implementation:
- Transformed NotebookLlaMa into enterprise multi-user NotebookLM clone
- Fixed critical data leakage with per-notebook pipeline isolation
- Built complete chat system with privacy and sharing
- Added background processing for documents and podcasts
- Implemented comprehensive sharing with 3-tier permissions
- Created admin dashboard with usage statistics

All Major Features:
 User authentication and multi-tenancy
 Multi-document notebooks (NotebookLM style)
 Per-notebook LlamaCloud pipeline isolation (SECURE!)
 Background document processing (queue system)
 Background podcast generation (queue system)
 Cross-document synthesis and analysis
 Multiple chat sessions per notebook
 Private chats by default, optional sharing
 Chat history persistence
 Rename/delete/share chat sessions
 3-tier permissions: Read, Write, Write+Share
 Custom podcast themes and prompts
 Voice selection (8 ElevenLabs voices)
 Podcast length control (5-30 minutes)
 Share management (add/remove users)
 Admin dashboard with cost tracking
 Professional UI with Montserrat typography
 Yellow branding (#FFC407)
 Logo integration

Database Schema (7 tables):
- users (authentication)
- notebooks (collections with dedicated pipelines)
- documents (PDFs)
- notebook_documents (many-to-many)
- document_summaries (AI analysis)
- chat_sessions (with privacy and sharing)
- chat_messages (conversation history)
- document_shares (permissions)
- background_tasks (async processing)

Security Features:
 Per-notebook pipeline isolation (no data leakage!)
 Private chats by default
 Granular sharing permissions
 Access control on all operations
 Password hashing with bcrypt
 SQL injection protection (ORM)

Performance:
 Background task queue (no blocking UI)
 Database connection pooling
 Navigate away during long operations
 Status tracking for all async tasks

Documentation:
- README.md (comprehensive install guide)
- ENTERPRISE_SETUP.md
- IMPLEMENTATION_SUMMARY.md
- SIMPLIFIED_PLAN.md
- CURRENT_STATUS.md
- CRITICAL_ISSUE.md (resolved!)

Files Created/Modified: 40+
Lines Added: 8000+
Total Cost: $78.39
Duration: 17+ hours

Production-Ready Features:
- Multi-user support
- Data isolation
- Background processing
- Admin monitoring
- Cost tracking
- Professional UI
- Complete documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 09:11:00 -04:00
DJP
aa66d14301 Fix chat share button text and complete chat system
Chat System Complete:
- Fixed function name conflict (chat → query_chat)
- Private chats show "🤝 Share" button (action to take)
- Shared chats show "🔒 Make Private" button (action to take)
- Button text now shows the ACTION not the current state
- New Chat button works perfectly
- Chat sessions listed in sidebar
- Multiple chats per notebook per user
- Rename chat functionality
- Delete chat functionality
- Share/unshare toggle
- Privacy by default

Features Working:
 Create new chat (fresh context)
 Switch between chat sessions
 Rename any chat
 Share chat with collaborators
 Unshare (make private again)
 Delete chat sessions
 View shared chats from others
 Chat history persistence
 Private by default
 Per-notebook pipeline isolation

Bug Fixes:
- Renamed chat() function to avoid shadowing by loop variable
- Fixed shared chat actions menu (⋮ now visible)
- Added extensive debug logging
- Better error handling in pipeline queries
- Fixed button text logic (show action, not state)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-01 20:43:10 -04:00
DJP
ccf82aef51 Rename chat() to query_chat() to avoid variable name conflict 2025-10-01 20:36:54 -04:00
DJP
b625664db0 Add extensive debug logging to chat functions 2025-10-01 20:35:15 -04:00
DJP
e6ff68b962 Add shared chat actions menu, rename functionality, and debug logging 2025-10-01 20:32:34 -04:00
DJP
33465ea337 Add better error handling and logging to pipeline query 2025-10-01 20:29:08 -04:00
DJP
2197405344 Add ChatSession imports to toggle and delete functions 2025-10-01 20:26:15 -04:00
DJP
26e03b1650 Fix chat session dict access (was trying to call object) 2025-10-01 20:24:11 -04:00
DJP
235a59ddb9 Fix chat history loading (corrupted code) 2025-10-01 20:21:38 -04:00
DJP
064e23127f Add ChatSession import and better error logging 2025-10-01 20:15:08 -04:00