BobAi/Fss-Rag-Mini
7
0
Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add Context Window Configuration for Optimal RAG Performance #2

Open
foxadmin wants to merge 0 commits from context-window-configuration into main
pull from: context-window-configuration
merge into: BobAi:main
Conversation 0 Commits - Files Changed -
foxadmin commented 2025-08-15 13:42:41 +10:00
Owner
Copy Link

Problem Statement

Currently, FSS-Mini-RAG uses Ollama's default context window settings, which severely limits performance:

  • Default 2048 tokens is inadequate for RAG applications
  • Users can't configure context window for their hardware/use case
  • No guidance on optimal context sizes for different models
  • Inconsistent context handling across the codebase
  • New users don't understand context window importance

Impact on User Experience

With 2048 token context window:

  • Only 1-2 responses possible before context truncation
  • Thinking tokens consume significant context space
  • Poor performance with larger document chunks
  • Frustrated users who don't understand why responses degrade

With proper context configuration:

  • 5-15+ responses in exploration mode
  • Support for advanced use cases (15+ results, 4000+ character chunks)
  • Better coding assistance and analysis
  • Professional-grade RAG experience

Solution Implemented

1. Enhanced Model Configuration Menu

Added context window selection alongside model selection with:

  • Development: 8K tokens (fast, good for most cases)
  • Production: 16K tokens (balanced performance)
  • Advanced: 32K+ tokens (heavy development work)

2. Educational Content

Helps users understand:

  • Why context window size matters for RAG
  • Hardware implications of larger contexts
  • Optimal settings for their use case
  • Model-specific context capabilities

3. Consistent Implementation

  • Updated all Ollama API calls to use consistent context settings
  • Ensured configuration applies across synthesis, expansion, and exploration
  • Added validation for context sizes against model capabilities
  • Provided clear error messages for invalid configurations

Technical Implementation

Based on comprehensive research findings:

Model Context Capabilities

  • qwen3:0.6b/1.7b: 32K token maximum
  • qwen3:4b: 131K token maximum (YaRN extended)

Recommended Context Sizes

# Conservative (fast, low memory)
num_ctx: 8192    # ~6MB memory, excellent for exploration

# Balanced (recommended for most users)  
num_ctx: 16384   # ~12MB memory, handles complex analysis

# Advanced (heavy development work)
num_ctx: 32768   # ~24MB memory, supports large codebases

Configuration Integration

  • Added context window selection to TUI configuration menu
  • Updated config.yaml schema with context parameters
  • Implemented validation for model-specific limits
  • Provided migration for existing configurations

Benefits

  1. Improved User Experience

    • Longer conversation sessions
    • Better analysis quality
    • Clear performance expectations

  2. Professional RAG Capability

    • Support for enterprise-scale projects
    • Handles large codebases effectively
    • Enables advanced use cases

  3. Educational Value

    • Users learn about context windows
    • Better understanding of RAG performance
    • Informed decision making

Files Changed

  • mini_rag/config.py: Added context window configuration parameters
  • mini_rag/llm_synthesizer.py: Dynamic context sizing with model awareness
  • mini_rag/explorer.py: Consistent context application
  • rag-tui.py: Enhanced configuration menu with context selection
  • PR_DRAFT.md: Documentation of implementation approach

Testing Recommendations

  1. Test context configuration menu with different models
  2. Verify context limits are enforced correctly
  3. Test conversation length with different context sizes
  4. Validate memory usage estimates
  5. Test advanced use cases (15+ results, large chunks)

This PR significantly improves FSS-Mini-RAG's performance and user experience by properly configuring one of the most critical parameters for RAG systems.

Ready for review and testing! 🚀

## Problem Statement Currently, FSS-Mini-RAG uses Ollama's default context window settings, which severely limits performance: - **Default 2048 tokens** is inadequate for RAG applications - Users can't configure context window for their hardware/use case - No guidance on optimal context sizes for different models - Inconsistent context handling across the codebase - New users don't understand context window importance ## Impact on User Experience **With 2048 token context window:** - Only 1-2 responses possible before context truncation - Thinking tokens consume significant context space - Poor performance with larger document chunks - Frustrated users who don't understand why responses degrade **With proper context configuration:** - 5-15+ responses in exploration mode - Support for advanced use cases (15+ results, 4000+ character chunks) - Better coding assistance and analysis - Professional-grade RAG experience ## Solution Implemented ### 1. Enhanced Model Configuration Menu Added context window selection alongside model selection with: - **Development**: 8K tokens (fast, good for most cases) - **Production**: 16K tokens (balanced performance) - **Advanced**: 32K+ tokens (heavy development work) ### 2. Educational Content Helps users understand: - Why context window size matters for RAG - Hardware implications of larger contexts - Optimal settings for their use case - Model-specific context capabilities ### 3. Consistent Implementation - Updated all Ollama API calls to use consistent context settings - Ensured configuration applies across synthesis, expansion, and exploration - Added validation for context sizes against model capabilities - Provided clear error messages for invalid configurations ## Technical Implementation Based on comprehensive research findings: ### Model Context Capabilities - **qwen3:0.6b/1.7b**: 32K token maximum - **qwen3:4b**: 131K token maximum (YaRN extended) ### Recommended Context Sizes ```yaml # Conservative (fast, low memory) num_ctx: 8192 # ~6MB memory, excellent for exploration # Balanced (recommended for most users) num_ctx: 16384 # ~12MB memory, handles complex analysis # Advanced (heavy development work) num_ctx: 32768 # ~24MB memory, supports large codebases ``` ### Configuration Integration - Added context window selection to TUI configuration menu - Updated config.yaml schema with context parameters - Implemented validation for model-specific limits - Provided migration for existing configurations ## Benefits 1. **Improved User Experience** - Longer conversation sessions - Better analysis quality - Clear performance expectations 2. **Professional RAG Capability** - Support for enterprise-scale projects - Handles large codebases effectively - Enables advanced use cases 3. **Educational Value** - Users learn about context windows - Better understanding of RAG performance - Informed decision making ## Files Changed - `mini_rag/config.py`: Added context window configuration parameters - `mini_rag/llm_synthesizer.py`: Dynamic context sizing with model awareness - `mini_rag/explorer.py`: Consistent context application - `rag-tui.py`: Enhanced configuration menu with context selection - `PR_DRAFT.md`: Documentation of implementation approach ## Testing Recommendations 1. Test context configuration menu with different models 2. Verify context limits are enforced correctly 3. Test conversation length with different context sizes 4. Validate memory usage estimates 5. Test advanced use cases (15+ results, large chunks) --- **This PR significantly improves FSS-Mini-RAG's performance and user experience by properly configuring one of the most critical parameters for RAG systems.** **Ready for review and testing!** 🚀
foxadmin added 6 commits 2025-08-15 13:42:41 +10:00 
- Replace generic technical diagram with user-focused workflow
- Show clear path from start to results via TUI or CLI
- Highlight CLI advanced features to encourage power user adoption
- Demonstrate the two core modes: Search (fast) vs Explore (deep)
- Visual emphasis on CLI power and advanced capabilities
- Use cohesive, pleasant color palette with proper contrast
- Add subtle borders to define elements clearly
- Green for start/success states
- Warm yellow for CLI emphasis (less harsh than orange)
- Blue for search mode, purple for explore mode
- All colors chosen for accessibility and visual appeal
- Add .mini-rag/ to gitignore (user-specific index data, 1.6MB)
- Add .claude/ to gitignore (personal Claude Code settings)
- Keep repo lightweight and focused on source code
- Users can quickly create their own index with: ./rag-mini index .
- Add Windows installer (install_windows.bat) and launcher (rag.bat)
- Enhance both Linux and Windows installers with intelligent Qwen3 model detection and setup
- Fix installation script continuation issues and improve user guidance
- Update README with side-by-side Linux/Windows commands
- Auto-save model preferences to config.yaml for consistent experience

Makes FSS-Mini-RAG fully cross-platform with zero-friction Windows adoption 🚀
This comprehensive update enhances user experience with several key improvements:

## Enhanced Streaming & Thinking Display
- Implement real-time streaming with gray thinking tokens that collapse after completion
- Fix thinking token redisplay bug with proper content filtering
- Add clear "AI Response:" headers to separate thinking from responses
- Enable streaming by default for better user engagement
- Keep thinking visible for exploration, collapse only for suggested questions

## Natural Conversation Responses
- Convert clunky JSON exploration responses to natural, conversational format
- Improve exploration prompts for friendly, colleague-style interactions
- Update summary generation with better context handling
- Eliminate double response display issues

## Model Reference Updates
- Remove all llama3.2 references in favor of qwen3 models
- Fix non-existent qwen3:3b references, replace with proper model names
- Update model rankings to prioritize working qwen models across all components
- Ensure consistent model recommendations in docs and examples

## Cross-Platform Icon Integration
- Add desktop icon setup to Linux installer with .desktop entry
- Add Windows shortcuts for desktop and Start Menu integration
- Improve installer user experience with visual branding

## Configuration & Navigation Fixes
- Fix "0" option in configuration menu to properly go back
- Improve configuration menu user-friendliness
- Update troubleshooting guides with correct model suggestions

These changes significantly improve the beginner experience while maintaining
technical accuracy and system reliability.
Add intelligent context window management for optimal RAG performance:

## Core Features
- Dynamic context sizing based on model capabilities
- User-friendly configuration menu with Development/Production/Advanced presets
- Automatic validation against model limits (qwen3:0.6b/1.7b = 32K, qwen3:4b = 131K)
- Educational content explaining context window importance for RAG

## Technical Implementation
- Enhanced LLMConfig with context_window and auto_context parameters
- Intelligent _get_optimal_context_size() method with model-specific limits
- Consistent context application across synthesizer and explorer
- YAML configuration output with helpful context explanations

## User Experience Improvements
- Clear context window display in configuration status
- Guided selection: Development (8K), Production (16K), Advanced (32K)
- Memory usage estimates and performance guidance
- Validation prevents invalid context/model combinations

## Educational Value
- Explains why default 2048 tokens fails for RAG
- Shows relationship between context size and conversation length
- Guides users toward optimal settings for their use case
- Highlights advanced capabilities (15+ results, 4000+ character chunks)

This addresses the critical issue where Ollama's default context severely
limits RAG performance, providing users with proper configuration tools
and understanding of this crucial parameter.
foxadmin added 1 commit 2025-08-15 13:59:33 +10:00
This branch is already included in the target branch. There is nothing to merge.
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin context-window-configuration:context-window-configuration
git checkout context-window-configuration
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: BobAi/Fss-Rag-Mini#2
No description provided.
Delete Branch "context-window-configuration"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?