Fss-Rag-Mini/docs/TESTING_PLAN.md

FSS-Mini-RAG Distribution Testing Plan

CRITICAL: This is a comprehensive testing plan for the new distribution system. Every stage must be completed and verified before deployment.

Overview

We've implemented a complete distribution overhaul with:

• One-line installers for Linux/macOS/Windows
• Multiple installation methods (uv, pipx, pip, zipapp)
• Automated wheel building via GitHub Actions
• PyPI publishing automation
• Cross-platform compatibility

This testing plan ensures everything works before we ship it.

---

Phase 1: Local Development Environment Testing

1.1 Virtual Environment Setup Testing

Objective: Verify our package works in clean environments

Test Environments:

• Python 3.8 in fresh venv
• Python 3.9 in fresh venv
• Python 3.10 in fresh venv
• Python 3.11 in fresh venv
• Python 3.12 in fresh venv

For each Python version:

# Test commands for each environment
python -m venv test_env_38
source test_env_38/bin/activate  # or test_env_38\Scripts\activate on Windows
python --version
pip install -e .
rag-mini --help
rag-mini init --help
rag-mini search --help
# Test basic functionality
mkdir test_project
echo "def hello(): print('world')" > test_project/test.py
rag-mini init -p test_project
rag-mini search -p test_project "hello function"
deactivate
rm -rf test_env_38 test_project

Success Criteria:

• Package installs without errors
• All CLI commands show help properly
• Basic indexing and search works
• No dependency conflicts

1.2 Package Metadata Testing

Objective: Verify pyproject.toml produces correct package metadata

Tests:

# Build source distribution and inspect metadata
python -m build --sdist
tar -tzf dist/*.tar.gz | grep -E "(pyproject.toml|METADATA)"
tar -xzf dist/*.tar.gz --to-stdout */METADATA

# Verify key metadata fields
python -c "
import pkg_resources
dist = pkg_resources.get_distribution('fss-mini-rag')
print(f'Name: {dist.project_name}')
print(f'Version: {dist.version}')  
print(f'Entry points: {list(dist.get_entry_map().keys())}')
"

Success Criteria:

• Package name is "fss-mini-rag"
• Console script "rag-mini" is registered
• Version matches pyproject.toml
• Author, license, description are correct
• Python version requirements are set

---

Phase 2: Build System Testing

2.1 Source Distribution Testing

Objective: Verify source packages build and install correctly

Tests:

# Clean build
rm -rf dist/ build/ *.egg-info/
python -m build --sdist

# Test source install in fresh environment
python -m venv test_sdist
source test_sdist/bin/activate
pip install dist/*.tar.gz
rag-mini --help
# Test actual functionality
mkdir test_src && echo "print('test')" > test_src/main.py
rag-mini init -p test_src
rag-mini search -p test_src "print statement"
deactivate && rm -rf test_sdist test_src

Success Criteria:

• Source distribution builds without errors
• Contains all necessary files
• Installs and runs correctly from source
• No missing dependencies

2.2 Wheel Building Testing

Objective: Test wheel generation and installation

Tests:

# Build wheel
python -m build --wheel

# Inspect wheel contents  
python -m zipfile -l dist/*.whl
python -m wheel unpack dist/*.whl
ls -la fss_mini_rag-*/

# Test wheel install
python -m venv test_wheel
source test_wheel/bin/activate
pip install dist/*.whl
rag-mini --version
which rag-mini
rag-mini --help
deactivate && rm -rf test_wheel

Success Criteria:

• Wheel builds successfully
• Contains correct package structure
• Installs faster than source
• Entry point is properly registered

2.3 Zipapp (.pyz) Building Testing

Objective: Test single-file zipapp distribution

Tests:

# Build zipapp
python scripts/build_pyz.py

# Test direct execution
python dist/rag-mini.pyz --help
python dist/rag-mini.pyz --version

# Test with different Python versions
python3.8 dist/rag-mini.pyz --help
python3.11 dist/rag-mini.pyz --help

# Test functionality
mkdir pyz_test && echo "def test(): pass" > pyz_test/code.py
python dist/rag-mini.pyz init -p pyz_test
python dist/rag-mini.pyz search -p pyz_test "test function"
rm -rf pyz_test

# Test file size and contents
ls -lh dist/rag-mini.pyz
python -m zipfile -l dist/rag-mini.pyz | head -20

Success Criteria:

• Builds without errors
• File size is reasonable (< 100MB)
• Runs with multiple Python versions
• All core functionality works
• No missing dependencies in zipapp

---

Phase 3: Installation Script Testing

3.1 Linux/macOS Install Script Testing

Objective: Test install.sh in various Unix environments

Test Environments:

• Ubuntu 20.04 (clean container)
• Ubuntu 22.04 (clean container)
• Ubuntu 24.04 (clean container)
• CentOS 7 (clean container)
• CentOS Stream 9 (clean container)
• macOS 12+ (if available)
• Alpine Linux (minimal test)

For each environment:

# Test script download and execution
curl -fsSL file://$(pwd)/install.sh > /tmp/test_install.sh
chmod +x /tmp/test_install.sh

# Test dry run capabilities (modify script for --dry-run flag)
/tmp/test_install.sh --dry-run

# Test actual installation
/tmp/test_install.sh

# Verify installation
which rag-mini
rag-mini --help
rag-mini --version

# Test functionality
mkdir install_test
echo "def example(): return 'hello'" > install_test/sample.py
rag-mini init -p install_test  
rag-mini search -p install_test "example function"

# Cleanup
rm -rf install_test /tmp/test_install.sh

Edge Case Testing:

# Test without curl
mv /usr/bin/curl /usr/bin/curl.bak 2>/dev/null || true
# Run installer (should fall back to wget or pip)
# Restore curl

# Test without wget  
mv /usr/bin/wget /usr/bin/wget.bak 2>/dev/null || true
# Run installer
# Restore wget

# Test with Python but no pip
# Test with old Python versions
# Test with no internet (local package test)

Success Criteria:

• Script downloads and runs without errors
• Handles missing dependencies gracefully
• Installs correct package version
• Creates working rag-mini command
• Provides clear user feedback
• Falls back properly (uv → pipx → pip)

3.2 Windows PowerShell Script Testing

Objective: Test install.ps1 in Windows environments

Test Environments:

• Windows 10 (PowerShell 5.1)
• Windows 11 (PowerShell 5.1)
• Windows Server 2019
• PowerShell Core 7.x (cross-platform)

For each environment:

# Download and test
Invoke-WebRequest -Uri "file://$(Get-Location)/install.ps1" -OutFile "$env:TEMP/test_install.ps1"

# Test execution policy handling
Get-ExecutionPolicy
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process

# Test dry run (modify script)
& "$env:TEMP/test_install.ps1" -DryRun

# Test actual installation
& "$env:TEMP/test_install.ps1"

# Verify installation
Get-Command rag-mini
rag-mini --help
rag-mini --version

# Test functionality
New-Item -ItemType Directory -Name "win_test"
"def windows_test(): return True" | Out-File -FilePath "win_test/test.py"
rag-mini init -p win_test
rag-mini search -p win_test "windows test"

# Cleanup
Remove-Item -Recurse -Force win_test
Remove-Item "$env:TEMP/test_install.ps1"

Edge Case Testing:

• Test without Python in PATH
• Test with Python 3.8-3.12
• Test restricted execution policy
• Test without admin rights
• Test corporate firewall scenarios

Success Criteria:

• Script runs without PowerShell errors
• Handles execution policy correctly
• Installs package successfully
• PATH is updated correctly
• Error messages are user-friendly
• Falls back properly (uv → pipx → pip)

---

Phase 4: GitHub Actions Workflow Testing

4.1 Local Workflow Testing

Objective: Test GitHub Actions workflow locally using act

Setup:

# Install act (GitHub Actions local runner)
# On macOS: brew install act
# On Linux: check https://github.com/nektos/act

# Test workflow syntax
act --list

# Test individual jobs
act -j build-wheels --dry-run
act -j build-zipapp --dry-run  
act -j test-installation --dry-run

Tests:

# Test wheel building job
act -j build-wheels

# Check artifacts
ls -la /tmp/act-* 

# Test zipapp building
act -j build-zipapp

# Test installation testing job
act -j test-installation

# Test release job (with dummy tag)
act push -e .github/workflows/test-release.json

Success Criteria:

• All jobs complete without errors
• Wheels are built for all platforms
• Zipapp is created successfully
• Installation tests pass
• Artifacts are properly uploaded

4.2 Fork Testing

Objective: Test workflow in a real GitHub environment

Setup:

1. Create a test fork of the repository
2. Enable GitHub Actions on the fork
3. Set up test PyPI token (TestPyPI)

Tests:

# Push changes to test branch
git checkout -b test-distribution
git push origin test-distribution

# Create test release
git tag v2.1.0-test
git push origin v2.1.0-test

# Monitor GitHub Actions:
# - Check all jobs complete
# - Download artifacts
# - Verify wheel contents  
# - Test zipapp download

Success Criteria:

• Workflow triggers on tag push
• All matrix builds complete
• Artifacts are uploaded
• Release is created with assets
• TestPyPI receives package (if configured)

---

Phase 5: Manual Installation Method Testing

5.1 uv Installation Testing

Test Environments: Linux, macOS, Windows

Tests:

# Fresh environment
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

# Test uv tool install (will fail until we publish)  
# For now, test with local wheel
uv tool install dist/fss_mini_rag-*.whl

# Verify installation
which rag-mini
rag-mini --help

# Test functionality
mkdir uv_test
echo "print('uv test')" > uv_test/demo.py
rag-mini init -p uv_test
rag-mini search -p uv_test "print statement"
rm -rf uv_test

# Test uninstall
uv tool uninstall fss-mini-rag

Success Criteria:

• uv installs cleanly
• Package installs via uv tool install
• Command is available in PATH
• All functionality works
• Uninstall works cleanly

5.2 pipx Installation Testing

Test Environments: Linux, macOS, Windows

Tests:

# Install pipx
python -m pip install --user pipx
python -m pipx ensurepath

# Test pipx install (local wheel for now)
pipx install dist/fss_mini_rag-*.whl

# Verify installation
pipx list
which rag-mini  
rag-mini --help

# Test functionality
mkdir pipx_test
echo "def pipx_demo(): pass" > pipx_test/code.py
rag-mini init -p pipx_test
rag-mini search -p pipx_test "pipx demo"
rm -rf pipx_test

# Test uninstall
pipx uninstall fss-mini-rag

Success Criteria:

• pipx installs without issues
• Package is isolated in own environment
• Command works globally
• No conflicts with system packages
• Uninstall is clean

5.3 pip Installation Testing

Test Environments: Multiple Python versions

Tests:

# Test with --user flag
pip install --user dist/fss_mini_rag-*.whl

# Verify PATH  
echo $PATH | grep -q "$(python -m site --user-base)/bin"
which rag-mini
rag-mini --help

# Test functionality
mkdir pip_test
echo "class PipTest: pass" > pip_test/example.py
rag-mini init -p pip_test
rag-mini search -p pip_test "PipTest class"
rm -rf pip_test

# Test uninstall
pip uninstall -y fss-mini-rag

Success Criteria:

• Installs correctly with --user
• PATH is configured properly
• No permission issues
• Works across Python versions
• Uninstall removes everything

---

Phase 6: End-to-End User Experience Testing

6.1 New User Experience Testing

Scenario: Complete beginner with no Python knowledge

Test Script:

# Start with fresh system (VM/container)
# Follow README instructions exactly

# Linux/macOS user
curl -fsSL https://raw.githubusercontent.com/fsscoding/fss-mini-rag/main/install.sh | bash

# Windows user  
# iwr https://raw.githubusercontent.com/fsscoding/fss-mini-rag/main/install.ps1 -UseBasicParsing | iex

# Follow quick start guide
rag-mini --help
mkdir my_project
echo "def hello_world(): print('Hello RAG!')" > my_project/main.py
echo "class DataProcessor: pass" > my_project/processor.py
rag-mini init -p my_project
rag-mini search -p my_project "hello function"
rag-mini search -p my_project "DataProcessor class"

Success Criteria:

• Installation completes without user intervention
• Clear, helpful output throughout
• rag-mini command is available immediately
• Basic workflow works as expected
• Error messages are user-friendly

6.2 Developer Experience Testing

Scenario: Python developer wanting to contribute

Test Script:

# Clone repository
git clone https://github.com/fsscoding/fss-mini-rag.git
cd fss-mini-rag

# Development installation
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

# Test development commands
make help
make dev-install
make test-dist
make build
make build-pyz

# Test local installation
pip install dist/*.whl
rag-mini --help

Success Criteria:

• Development setup is straightforward
• Makefile commands work correctly
• Local builds install properly
• All development tools function

6.3 Advanced User Testing

Scenario: Power user with custom requirements

Test Script:

# Test zipapp usage
wget https://github.com/fsscoding/fss-mini-rag/releases/latest/download/rag-mini.pyz
python rag-mini.pyz --help

# Test with large codebase
git clone https://github.com/django/django.git test_django
python rag-mini.pyz init -p test_django
python rag-mini.pyz search -p test_django "model validation"

# Test server mode  
python rag-mini.pyz server -p test_django
curl http://localhost:7777/health

# Clean up
rm -rf test_django rag-mini.pyz

Success Criteria:

• Zipapp handles large codebases
• Performance is acceptable
• Server mode works correctly
• All advanced features function

---

Phase 7: Performance and Edge Case Testing

7.1 Performance Testing

Objective: Ensure installation and runtime performance is acceptable

Tests:

# Installation speed testing
time curl -fsSL https://raw.githubusercontent.com/fsscoding/fss-mini-rag/main/install.sh | bash

# Package size testing
ls -lh dist/
du -sh .venv/

# Runtime performance
time rag-mini init -p large_project/
time rag-mini search -p large_project/ "complex query"

# Memory usage
rag-mini server &
ps aux | grep rag-mini
# Monitor memory usage during indexing/search

Success Criteria:

• Installation completes in < 5 minutes
• Package size is reasonable (< 50MB total)
• Indexing performance meets expectations
• Memory usage is acceptable

7.2 Edge Case Testing

Objective: Test unusual but possible scenarios

Tests:

# Network issues
# - Simulate slow connection
# - Test offline scenarios  
# - Test corporate firewalls

# System edge cases
# - Very old Python versions
# - Systems without pip
# - Read-only file systems
# - Limited disk space

# Unicode and special characters
mkdir "测试项目"
echo "def 函数名(): pass" > "测试项目/代码.py"
rag-mini init -p "测试项目"
rag-mini search -p "测试项目" "函数"

# Very large files
python -c "print('# ' + 'x'*1000000)" > large_file.py
rag-mini init -p .
# Should handle gracefully

# Concurrent usage
rag-mini server &
for i in {1..10}; do
    rag-mini search "test query $i" &
done
wait

Success Criteria:

• Graceful degradation with network issues
• Clear error messages for edge cases
• Handles Unicode correctly
• Doesn't crash on large files
• Concurrent access works properly

---

Phase 8: Security Testing

8.1 Install Script Security

Objective: Verify install scripts are secure

Tests:

# Check install.sh
shellcheck install.sh
bandit -r install.sh (if applicable)

# Verify HTTPS usage
grep -n "http://" install.sh  # Should only be for localhost
grep -n "curl.*-k" install.sh  # Should be none
grep -n "wget.*--no-check" install.sh  # Should be none

# Check PowerShell script
# Run PowerShell security analyzer if available

Success Criteria:

• No shell script vulnerabilities
• Only HTTPS downloads (except localhost)
• No certificate verification bypasses
• Input validation where needed
• Clear error messages without info leakage

8.2 Package Security

Objective: Ensure distributed packages are secure

Tests:

# Check for secrets in built packages
python -m zipfile -l dist/*.whl | grep -i -E "(key|token|password|secret)"
strings dist/rag-mini.pyz | grep -i -E "(key|token|password|secret)"

# Verify package signatures (when implemented)
# Check for unexpected executables in packages

Success Criteria:

• No hardcoded secrets in packages
• No unexpected executables
• Package integrity is verifiable
• Dependencies are from trusted sources

---

Phase 9: Documentation and User Support Testing

9.1 Documentation Accuracy Testing

Objective: Verify all documentation matches reality

Tests:

# Test every command in README
# Test every code example
# Verify all links work
# Check screenshots are current

# Test error scenarios mentioned in docs
# Verify troubleshooting sections

Success Criteria:

• All examples work as documented
• Links are valid and up-to-date
• Screenshots reflect current UI
• Error scenarios are accurate

9.2 Support Path Testing

Objective: Test user support workflows

Tests:

• GitHub issue templates work
• Error messages include helpful information
• Common problems have clear solutions
• Contact information is correct

---

Phase 10: Release Readiness

10.1 Pre-Release Checklist

• All tests from Phases 1-9 pass
• Version numbers are consistent
• Changelog is updated
• Documentation is current
• Security review complete
• Performance benchmarks recorded
• Backup plan exists for rollback

10.2 Release Testing

TestPyPI Release:

# Upload to TestPyPI first
python -m twine upload --repository testpypi dist/*

# Test installation from TestPyPI
pip install --index-url https://test.pypi.org/simple/ fss-mini-rag

Success Criteria:

• TestPyPI upload succeeds
• Installation from TestPyPI works
• All functionality works with TestPyPI package

10.3 Production Release

Only after TestPyPI success:

# Create GitHub release
git tag v2.1.0
git push origin v2.1.0

# Monitor automated workflows
# Test installation after PyPI publication
pip install fss-mini-rag

---

Testing Tools and Infrastructure

Required Tools

• Docker (for clean environment testing)
• act (for local GitHub Actions testing)
• shellcheck (for bash script analysis)
• Various Python versions (3.8-3.12)
• Windows VM/container access
• macOS testing environment (if possible)

Test Data

• Sample codebases of various sizes
• Unicode test files
• Edge case files (very large, empty, binary)
• Network simulation tools

Monitoring

• Performance benchmarks
• Error rate tracking
• User feedback collection
• Download/install statistics

---

Conclusion

This testing plan is comprehensive but necessary. Each phase builds on the previous ones, and skipping phases risks shipping broken functionality to users.

Estimated Timeline: 3-5 days for complete testing Risk Level: HIGH if phases are skipped Success Criteria: 100% of critical tests must pass before release

The goal is to ship a distribution system that "just works" for every user, every time. This level of testing ensures we achieve that goal.