SUITE5 — FAILURE-AWARE COMPETENCE COUNCIL ========================================= Purpose ------- Suite5 starts from Suite4's strongest experiment: the four-expert Council (committee + global HGB + recent HGB + relational expert). It changes the competence problem rather than adding another base specialist. Main changes ------------ 1. Keeps an exact Suite4-style symmetric-regret Council as a counterfactual. 2. Adds a failure-aware competence target that: - rewards rescuing failures missed by the committee, - more strongly penalizes losing already-detected failures, - rewards removal of false positives, - penalizes newly introduced false positives. 3. Upweights failures, committee-missed failures, and rows near the warning boundary during competence fitting. 4. Adds unlabeled boundary/rank features to competence inputs. 5. Tests raw vs rank-preserving aligned expert score spaces. 6. Tests broad temperature values (.15,.25,.5,1,2,4) to allow specialization, but does not force sharp weights. Equivalent candidates prefer smoother gates. 7. Restores full production evaluation: - pooled-development F2 threshold, - frozen holdout threshold transfer, - drift-aware warning rate, - precision/recall/F1/F2, - warning rate, - TN/FP/FN/TP confusion matrices, - equal-20%-budget metrics. Key outputs ----------- _summary.csv _threshold_policy.csv # one frozen development policy per model _warning_metrics.csv _fold_N_metrics.csv _fold_N_scores.csv _fold_N_policy_grid.csv _fold_N_selected_policies.csv _fold_N_weight_diagnostics.csv _fold_N_competence_confusion.csv _fold_N_history_regimes.csv Important model columns/counterfactuals --------------------------------------- committee uniform_raw legacy_council # same family, Suite5 combined selector suite4_legacy_exact # Suite4-style selection counterfactual failure_aware_council suite5_final NON_DEPLOYABLE_best_expert_oracle Run --- chmod +x run_suite5.sh DEVICE=auto ./run_suite5.sh \ /root/disk_3tb/earlywarning/data/splits/expanding \ /root/disk_3tb/earlywarning/30072026/results/suite5 \ khoa_cong_nghe_thong_tin Reuse the already-computed Suite4 canonical cache (recommended if it is still present and was generated with the same 500/250/5 settings): CACHE_DIR_OVERRIDE=/root/disk_3tb/earlywarning/30072026/results/suite4/cache \ DEVICE=auto ./run_suite5.sh \ /root/disk_3tb/earlywarning/data/splits/expanding \ /root/disk_3tb/earlywarning/30072026/results/suite5 \ khoa_cong_nghe_thong_tin Interpretation -------------- Do not judge the new gate only by whether its weights are sharper. Inspect: - AP/AUC and F2@20, - frozen-threshold and drift-aware F2, - rescued failures vs lost existing TPs, - mean weight on failures vs normals vs missed failures, - normalized entropy/effective number of experts, - best-expert diagnostic ceiling vs achieved Council performance. A near-uniform gate is not a failure if it produces stronger forward metrics. A sharper gate is only useful if its extra specialization transfers temporally. NEW RESEARCH BRANCH — CROSS-FACULTY RESIDUAL MoE ================================================= Suite5 now also contains a separate breakthrough research branch that does NOT modify the original failure-aware Council implementation. See: README_cross_faculty_residual_moe.txt prepare_cross_faculty_cache.py cross_faculty_common.py cross_faculty_residual_moe.py train_cross_faculty_residual_moe.py cross_faculty_eval.py collect_reference_metrics.py run_cross_faculty_residual_moe.sh This branch keeps the canonical Committee frozen and replaces hand-designed Global/Recent/Relational specialists with jointly discovered cross-faculty residual specialists plus an explicit NULL/Committee route.