LLM now decides which new categories should be added to persistent cache
for future mailbox runs vs temporary (run-only) categories.
ENHANCED LLM REVIEW:
- New field: "cache_worthy" (true/false) for each "new" category
- LLM judges: "Is this category useful across different mailboxes?"
- Examples:
- "Customer Support" → cache_worthy: true (universal)
- "Project X Updates" → cache_worthy: false (mailbox-specific)
CACHE EVOLUTION:
- cache_worthy=true → Added to persistent cache for future runs
- cache_worthy=false → Used for current run only, not cached
- First run (empty cache) → All categories treated as cache-worthy
- LLM reasoning logged for transparency
INTELLIGENT GROWTH:
- Cache grows organically with high-quality, reusable categories
- Prevents pollution with mailbox-specific categories
- Maintains cross-mailbox consistency while allowing natural evolution
- LLM balances: consistency (snap existing) vs expansion (add worthy)
SINGLE LLM CALL EFFICIENCY:
- Same ~4 second LLM call now handles:
1. Snap vs new decision
2. Cache persistence decision
3. Reasoning for both
- No additional overhead for cache evolution
Result: Cache evolves intelligently over time, collecting universally
useful categories while filtering out temporary/specific ones.
CategoryCache now uses Ollama embeddings + cosine similarity for
true semantic category matching instead of weak keyword overlap.
Changes:
- src/calibration/category_cache.py: Use embedder.embeddings() API
- Calculate embeddings for discovered and cached category descriptions
- Compute cosine similarity between embedding vectors
- Fall back to partial name matching if embeddings unavailable
- Error handling with graceful degradation
- src/calibration/workflow.py: Pass feature_extractor.embedder
- Provide Ollama client to CalibrationAnalyzer
- Enables semantic matching during cache snap
- src/calibration/llm_analyzer.py: Accept embedding_model parameter
- Forward embedder to CategoryCache constructor
Test Results (embedding-based vs keyword):
- "Training Materials" → "Training": 0.72 (was 0.15)
- "Team Updates" → "Work Communication": 0.62 (was 0.24)
- "System Alerts" → "Technical": 0.63 (was 0.12)
- "Meeting Invitations" → "Meetings": 0.75+ (exact match)
Semantic matching now properly identifies similar categories based
on meaning rather than superficial word overlap.
Category Cache System (src/calibration/category_cache.py):
- Persistent storage of discovered categories across mailbox runs
- Semantic matching to snap new categories to existing ones
- Usage tracking for category popularity
- Configurable similarity threshold and new category limits
- JSON-based cache with metadata (created, last_seen, email counts)
Discovery Improvements (src/calibration/llm_analyzer.py):
- Calculate batch statistics: sender domains, recipient counts,
attachments, subject lengths, common keywords
- Add statistics to LLM discovery prompt for better decisions
- Integrate CategoryCache into CalibrationAnalyzer
- 3-step workflow: Discover → Consolidate → Snap to Cache
Consolidation Improvements:
- Add cached categories as hints in consolidation prompt
- LLM prefers snapping to established categories
- Maintains cross-mailbox consistency while allowing new categories
Configuration Parameters:
- use_category_cache: Enable/disable caching (default: true)
- cache_similarity_threshold: Min similarity for snap (default: 0.7)
- cache_allow_new: Allow new categories (default: true)
- cache_max_new: Max new categories per run (default: 3)
- category_cache_path: Custom cache location
Result: Consistent category sets across different mailboxes
with intelligent discovery of new categories when appropriate.
Both discovery and consolidation prompts now explain:
- What the system does (train ML classifier for auto-sorting)
- What makes good categories (broad, timeless, learnable)
- Why this matters (user needs, ML training requirements)
- How to think about the task (user-focused, functional)
Discovery prompt changes:
- Explains goal of identifying natural categories for ML training
- Lists guidelines for good categories (broad, user-focused, learnable)
- Provides concrete examples of functional categories
- Emphasizes PURPOSE over topic
Consolidation prompt changes:
- Explains full system context (LightGBM, auto-labeling, user search)
- Defines what makes categories effective for ML and users
- Provides user-centric thinking framework
- Emphasizes reusability and timelessness
Prompts now give the brilliant 8b model proper context to deliver
excellent category decisions instead of lazy generic categorization.
Enhanced _consolidate_categories() with comprehensive validation:
- Edge case guards: Skip if ≤5 categories or no labels
- Parameter validation: Clamp ranges for all config values
- 5-stage validation after LLM response:
1. Structure check (valid dicts)
2. Reduction check (consolidation must reduce count)
3. Target compliance (soft 50% overage limit)
4. Complete mapping (all old categories mapped)
5. Valid targets (all mappings point to existing categories)
- Auto-repair for common LLM failures:
- Unmapped categories → map to first consolidated category
- Invalid mapping targets → create missing categories
- Failed updates → log with details
- Fallback consolidation using top-N by count
- Triggered on JSON parse errors, validation failures
- Heuristic-based, no LLM required
- Guarantees output even if LLM fails
All error paths now have proper handling and logging.
Root cause: Pre-trained model was loading successfully, causing CLI to skip
calibration entirely. System went straight to classification with 35% model.
Changes:
- config: Set calibration_model to qwen3:8b-q4_K_M (larger model for better instruction following)
- cli: Create separate calibration_llm provider with 8b model
- llm_analyzer: Improved prompt to force exact email ID copying
- workflow: Merge discovered categories with predefined ones
- workflow: Add detailed error logging for label mismatches
- ml_classifier: Fixed model path checking (was checking None parameter)
- ml_classifier: Add dual API support (sklearn predict_proba vs LightGBM predict)
- ollama: Fixed model list parsing (use m.model not m.get('name'))
- feature_extractor: Switch to Ollama embeddings (instant vs 90s load time)
Result: Calibration now runs and generates 16 categories + 50 labels correctly.
Next: Investigate calibration sampling to reduce overfitting on small samples.
PHASE 9: Processing Pipeline & Queue Management (bulk_processor.py)
- BulkProcessor class for batch processing with checkpointing
- ProcessingCheckpoint: Save/resume state for resumable processing
- Handles batches with periodic checkpoints every N emails
- Tracks completed, queued_for_llm, and failed emails
- Progress callbacks for UI integration
PHASE 10: Calibration System (sampler.py, llm_analyzer.py)
- EmailSampler: Stratified and random sampling
- Stratifies by sender domain type for representativeness
- CalibrationAnalyzer: Use LLM to discover natural categories
- Batched analysis to control LLM load
- Maps discovered categories to universal schema
PHASE 11: Export & Reporting (exporter.py)
- ResultsExporter: Export to JSON, CSV, organized by category
- ReportGenerator: Generate human-readable text reports
- Category statistics and method breakdown
- Accuracy metrics and processing time tracking
PHASE 13: Enron Dataset Parser (enron_parser.py)
- Parses Enron maildir format into Email objects
- Handles multipart emails and attachments
- Date parsing with fallback for malformed dates
- Ready to train mock model on real data
PHASE 14: Main Orchestration (orchestration.py)
- EmailSorterOrchestrator: Coordinates entire pipeline
- 4-phase workflow: Calibration → Bulk → LLM → Export
- Lazy initialization of components
- Progress tracking and timing
- Full pipeline runner with resume support
Components Now Available:
✅ Sampling (stratified and random)
✅ Calibration (LLM-driven category discovery)
✅ Bulk processing (with checkpointing)
✅ LLM review (batched)
✅ Export (JSON, CSV, by category)
✅ Reporting (text summaries)
✅ Enron parsing (ready for training)
✅ Full orchestration (4 phases)
What's Left (Phases 15-16):
- E2E pipeline tests
- Integration test with Enron data
- Setup.py and wheel packaging
- Deployment documentation
Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
- Setup virtual environment and install all dependencies
- Implemented modular configuration system (YAML-based)
- Created logging infrastructure with rich formatting
- Built email data models (Email, Attachment, ClassificationResult)
- Implemented email provider abstraction with stubs:
* MockProvider for testing
* Gmail provider (credentials required)
* IMAP provider (credentials required)
- Implemented feature extraction pipeline:
* Semantic embeddings (sentence-transformers)
* Hard pattern detection (20+ patterns)
* Structural features (metadata, timing, attachments)
- Created ML classifier framework with MOCK Random Forest:
* Mock uses synthetic data for testing only
* Clearly labeled as test/development model
* Placeholder for real LightGBM training at home
- Implemented LLM providers:
* Ollama provider (local, qwen3:1.7b/4b support)
* OpenAI-compatible provider (API-based)
* Graceful degradation when LLM unavailable
- Created adaptive classifier orchestration:
* Hard rules matching (10%)
* ML classification with confidence thresholds (85%)
* LLM review for uncertain cases (5%)
* Dynamic threshold adjustment
- Built CLI interface with commands:
* run: Full classification pipeline
* test-config: Config validation
* test-ollama: LLM connectivity
* test-gmail: Gmail OAuth (when configured)
- Created comprehensive test suite:
* 23 unit and integration tests
* 22/23 passing
* Feature extraction, classification, end-to-end workflows
- Categories system with 12 universal categories:
* junk, transactional, auth, newsletters, social, automated
* conversational, work, personal, finance, travel, unknown
Status:
- Framework: 95% complete and functional
- Mocks: Clearly labeled, transparent about limitations
- Tests: Passing, validates integration
- Ready for: Real data training when Enron dataset available
- Next: Home setup with real credentials and model training
This build is production-ready for framework but NOT for accuracy.
Real ML model training, Gmail OAuth, and LLM will be done at home
with proper hardware and real inbox data.
Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
