FSSCoding
53174a34eb
Project Reorganization: - Created docs/ directory and moved all documentation - Created scripts/ directory for shell scripts - Created scripts/experimental/ for research scripts - Updated .gitignore for new structure - Updated README.md with MVP status and new structure New Features: - Category verification system (verify_model_categories) - --verify-categories flag for mailbox compatibility check - --no-llm-fallback flag for pure ML classification - Trained model saved in src/models/calibrated/ Threshold Optimization: - Reduced default threshold from 0.75 to 0.55 - Updated all category thresholds to 0.55 - Reduces LLM fallback rate by 40% (35% -> 21%) Documentation: - SYSTEM_FLOW.html - Complete system architecture - VERIFY_CATEGORIES_FEATURE.html - Feature documentation - LABEL_TRAINING_PHASE_DETAIL.html - Calibration breakdown - FAST_ML_ONLY_WORKFLOW.html - Pure ML guide - PROJECT_STATUS_AND_NEXT_STEPS.html - Roadmap - ROOT_CAUSE_ANALYSIS.md - Bug fixes MVP Status: - 10k emails in 4 minutes, 72.7% accuracy, 0 LLM calls - LLM-driven category discovery working - Embedding-based transfer learning confirmed - All model paths verified and working
31 lines
767 B
Bash
Executable File
31 lines
767 B
Bash
Executable File
#!/bin/bash
|
|
# Test ML performance without LLM fallback using trained model
|
|
|
|
set -e
|
|
|
|
echo "=========================================="
|
|
echo "ML-ONLY TEST (No LLM Fallback)"
|
|
echo "=========================================="
|
|
echo ""
|
|
echo "Using model: src/models/calibrated/classifier.pkl"
|
|
echo "Testing on: 1000 emails"
|
|
echo ""
|
|
|
|
# Activate venv
|
|
if [ -z "$VIRTUAL_ENV" ]; then
|
|
source venv/bin/activate
|
|
fi
|
|
|
|
# Run classification with trained model, NO LLM fallback
|
|
python -m src.cli run \
|
|
--source enron \
|
|
--limit 1000 \
|
|
--output ml_only_test/ \
|
|
--no-llm-fallback \
|
|
2>&1 | tee ml_only_test.log
|
|
|
|
echo ""
|
|
echo "=========================================="
|
|
echo "Test complete. Check ml_only_test.log"
|
|
echo "=========================================="
|