SUITE11 — OPPORTUNITY-AWARE LATENT-DOMAIN SPECIALISTS ====================================================== Purpose ------- Suite11 tests whether chronology-safe historical OOF correction opportunities can be compressed into latent expert domains that remain predictable from observable pre-outcome features. It directly follows Suite9 (useful diversity but poor prospective routing) and Suite10 (prospective semantic domains but weak/unstable specialist advantage). Core design ----------- 1. Reuse the existing Suite5 cross-faculty cache and Suite7 backbone/checkpoints. 2. Build OOF opportunity signatures from the three committee experts + Committee: base-expert utility, Committee BCE/signed error, missed-failure severity, false-alarm severity, within-faculty/block Committee rank, disagreement, utility dispersion, entropy. 3. Per faculty/temporal block, lower-opportunity rows become NULL; higher-opportunity rows are clustered into K latent domains (default K=4 for controlled Suite7 checkpoint reuse). 4. Train a prospective MLP router on observable X only to predict NULL + latent domains. 5. Train Suite7-initialized residual experts using soft latent-domain responsibilities. 6. Select a global specialist-authority alpha on chronology-safe historical validation. 7. Use Suite11 only when the newest historical selection block beats exact Suite7; otherwise use exact Suite7 fallback. Fair comparison/reporting ------------------------- The trainer loads, where supplied, the actual per-fold score files from: - Committee / Suite5 shared - Suite7 - Suite8 - Suite9 candidate/final - Suite10 candidate/final - final_train_v2 classifier - Suite4 Competence Council Every available model is then re-evaluated with the SAME Suite11 evaluator: - fold-wise + mean-within-fold AP/AUC - equal-budget P/R/F1/F2 at 10/15/20/25% - development-tuned fixed threshold P/R/F1/F2 + full confusion matrix - drift-aware policy P/R/F1/F2 + full confusion matrix - TP/FN/FP/TN totals For the primary faculty (default IT), detailed confusion matrices are printed for the main comparators automatically. Use --verbose_operational to print them for every faculty/model. Suite11-specific diagnostics ---------------------------- - latent opportunity cluster centers - NULL/domain prevalence and failure prevalence - router balanced accuracy, macro-F1, NMI and confusion matrix - target hindsight router agreement (DIAGNOSTIC ONLY; never used for fitting/selection) - expert in-domain AP uplift vs Committee - expert residual correlation - specialist authority / NULL mass - 20% budget rescued failures, lost failures, removed/added false alarms - hindsight expert oracle AP/AUC and oracle gap - chronological deployment fallback decision Important files produced ------------------------ suite11_fold_metrics.csv suite11_holdout_mean_fold.csv suite11_holdout_pooled_raw.csv suite11_warning_metrics.csv suite11_macro_model_comparison.csv suite11_training_summary.csv suite11_latent_domain_diagnostics.csv suite11_residual_correlations.csv suite11_top20_transitions.csv fold_X/*_scores.csv fold_X/opportunity_domain_centers.csv fold_X/target_router_hindsight_confusion.csv fold_X/suite11_fold_summary.json The target outcome is used only after target scoring for diagnostic files. It never affects training, latent-domain discovery, routing, alpha selection, threshold tuning, or deployment.