SUITE9 — COMPETITIVE RESPONSIBILITY RESIDUAL MOE ================================================ Research question ----------------- Can genuinely different residual specialists emerge when training rows are assigned sparse expert responsibilities, instead of sending essentially the same prediction gradient to every expert? Suite9 is the direct follow-up to Suite8. Suite8 proved that different latent views can be learned, but the temporally selected solution stayed close to expert collapse. Suite9 therefore changes the source of specialization from "different information" to "different responsibility". Architecture ------------ The faculty-specific Committee remains frozen. Suite9 keeps the Suite7 shared encoder, faculty adapters, residual experts, NULL route, and Suite8-style latent expert views. For row i and expert k: l_i0 = BCE(y_i, q_i) p_ik = sigmoid(logit(q_i) + delta_ik) l_ik = BCE(y_i, p_ik) u_ik = l_i0 - l_ik NULL has utility 0. Experts with utility <= --min_expert_utility are ineligible. A sparse softmax is applied only to the best --responsibility_top_k routes among NULL + eligible experts. The resulting responsibilities are DETACHED before expert updates. Training has three phases: Phase 1 — Specialist discovery * Suite7 faculty embedding / shared encoder / faculty adapters / router frozen. * Only residual experts + latent-view ownership train. * Each expert receives gradient only in proportion to its sparse responsibility. * Responsibility assignment is deterministic: dropout is disabled while the counterfactual utilities are computed. Phase 2 — Router distillation * Experts / views / representation frozen. * Only the router trains. * Target = sparse competitive responsibility distribution from the frozen experts. * Router model selection is based on deployable AP + F2@20 versus Suite7, not versus Committee. Phase 3 — Optional joint refinement * Disabled by default (--joint_refine_epochs 0). * If enabled, experts/views/router refine with low LR using detached responsibility targets; it must outperform the Phase-2 model and Suite7 on temporal validation. Exact Suite7 fallback --------------------- Suite9 never forces a new model to deploy. For each fold, if the best deployable Suite9 candidate does not beat the temporally reproduced Suite7 baseline by --min_selection_improvement, the Suite9 FINAL score is exactly the saved Suite7 score for that target fold. The unselected Suite9 candidate is still saved, so research diagnostics remain available even when deployment falls back. Chronology / leakage handling ----------------------------- For temporal model selection, Suite9 DOES NOT initialize from the final Suite7 checkpoint, because that checkpoint was refit using all historical rows, including Suite9's validation blocks. Instead Suite9: 1. reproduces Suite7 from scratch on the chronological training subset for the selected Suite7 epoch count; 2. performs Suite9 discovery/router selection against that validation-safe baseline; 3. only after epoch counts/fallback are selected, initializes from the real fitted Suite7 checkpoint and refits Suite9 on all historical rows. Holdout labels are never used to select discovery epochs, router epochs, joint-refine epochs, fallback, fixed thresholds, or drift-aware warning ratios. Oracle responsibility scores use labels and are DIAGNOSTIC ONLY. Required existing artifacts --------------------------- No Committee cache rebuild is required. Default paths used in the recommended command: Suite5 cross-faculty cache: /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache Suite5 shared results: /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe Suite7 results/checkpoints: /root/disk_3tb/earlywarning/30072026/results/suite7_shared_private_adapter Suite8 results (read-only comparison; optional but recommended): /root/disk_3tb/earlywarning/30072026/results/suite8_true_specialist Recommended run --------------- cd /root/disk_3tb/earlywarning/30072026/suite9 mkdir -p /root/disk_3tb/earlywarning/30072026/results/suite9_competitive_responsibility mkdir -p /root/disk_3tb/earlywarning/30072026/log python3 train_suite9_competitive_responsibility.py \ --cache_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \ --shared_results_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe \ --suite7_results_dir /root/disk_3tb/earlywarning/30072026/results/suite7_shared_private_adapter \ --suite8_results_dir /root/disk_3tb/earlywarning/30072026/results/suite8_true_specialist \ --models_dir /root/disk_3tb/earlywarning/30072026/results/suite9_competitive_responsibility \ --faculties all \ --dev_folds 1,2,3,4,5 \ --holdout_folds 6,7 \ --seed 42 \ --device auto \ --responsibility_temperature 0.10 \ --responsibility_top_k 2 \ --min_expert_utility 0.0 \ --discovery_epochs 8 \ --discovery_min_epochs 2 \ --discovery_patience 4 \ --discovery_lr 0.0005 \ --router_epochs 12 \ --router_min_epochs 2 \ --router_patience 5 \ --router_lr 0.001 \ --joint_refine_epochs 0 \ --view_temperature 1.0 \ --view_strength 0.50 \ --view_entropy_lambda 0.002 \ --view_balance_lambda 0.01 \ --batch_size 1024 \ --predict_batch_size 4096 \ --val_blocks 2 \ --sampling_beta 0.9999 \ --macro_f2_weight 1.0 \ --min_selection_improvement 0.0001 \ --threshold_beta 2.0 \ --min_precision 0.15 \ --max_warning_ratio 2.5 \ --min_flags 5 \ > /root/disk_3tb/earlywarning/30072026/log/suite9_competitive_responsibility.log 2>&1 Monitor: tail -f /root/disk_3tb/earlywarning/30072026/log/suite9_competitive_responsibility.log Why these defaults? ------------------- * top_k=2: sparse enough to create ownership, but less brittle than hard top-1. * responsibility_temperature=0.10: sharper than the earlier capability softmax without turning tiny utility differences into completely hard assignments. * min_expert_utility=0: an expert must actually beat Committee to be eligible. * view entropy lambda is 10x smaller than Suite8: view diversity should now emerge mainly because experts own different rows, not because we force masks apart. * joint refinement is disabled for the first experiment so D1 cleanly tests "responsibility competition -> specialist discovery -> router distillation". Important console printings --------------------------- DISCOVERY phase prints: oracle_sel : diagnostic oracle responsibility score vs Suite7 validation medDeltaAP : median faculty oracle AP gain vs Suite7 medDeltaF2 : median faculty oracle F2@20 gain vs Suite7 NULLresp : mean responsibility mass assigned to Committee/NULL RespH : normalized responsibility entropy Rcorr : residual expert correlation OwnH / ViewCos : latent ownership/view diversity ROUTER phase prints: deploy_sel : deployable learned-router score vs Suite7 validation Acc : top-1 agreement with sparse responsibility target targetKL : KL(target responsibility || learned router) Fold conclusion prints: selected discovery/router/joint epochs validation improvement vs Suite7 SUITE9 or EXACT SUITE7 FALLBACK Per faculty TARGET print: AP for Committee / Suite5 Shared / Suite7 / Suite8 / Suite9 candidate / Suite9 final F2@20 for Suite7 / candidate / final OracleAP responsibility NULL mass router top-1 responsibility accuracy residual correlation deploy/fallback state Main output files ----------------- Global: suite9_comparison.csv suite9_fold_metrics.csv suite9_holdout_pooled.csv suite9_warning_metrics.csv suite9_threshold_policies.csv suite9_training_summary.csv suite9_responsibility_diagnostics.csv suite9_expert_region_diagnostics.csv suite9_router_agreement.csv suite9_oracle_gap.csv suite9_transfer_at20.csv suite9_residual_correlations.csv suite9_official_comparison.csv (only when --reference_metrics is given) Per fold: phase_training_history.csv competitive_responsibility_model.pt suite9_scaler.npz fold_metrics.csv responsibility_diagnostics.csv expert_region_diagnostics.csv router_agreement.csv oracle_gap.csv transfer_at20.csv residual_correlations.csv adapter_parameter_norms.csv _scores.csv How to interpret the critical diagnostics ----------------------------------------- 1. Specialist quality Check responsibility mass and expert_region_diagnostics.csv. A useful specialist should have materially higher mean utility on its assigned rows than on unassigned rows, with nontrivial assigned failure support. 2. Responsibility collapse If one expert owns almost everything, competitive learning collapsed to a single residual model. If NULL owns almost everything, no specialist reliably improves Committee. 3. Router bottleneck suite9_oracle_gap.csv separates expert quality from gate quality. Large Oracle AP/F2 gain + small learned gain + low router accuracy: experts exist, router is the bottleneck. Small Oracle gain: experts themselves are still the bottleneck. 4. Early-warning transitions suite9_transfer_at20.csv reports rescued failures, lost failures, removed FP and added FP relative to Suite7 at exactly 20% warning budget. 5. Deployment safety suite9_final is the score to use for fair deployable comparison. suite9_candidate is the unprotected architectural candidate. suite9_oracle is label-informed diagnostic only and MUST NOT be reported as a deployable model. Reference V2 / Council metrics ------------------------------ The copied collect_reference_metrics.py can be used exactly as in Suite7/8. Pass the resulting CSV with: --reference_metrics /path/to/v2_council_reference.csv and Suite9 will additionally write suite9_official_comparison.csv. Primary success criteria ------------------------ This experiment succeeds mechanistically only if we see ALL of the following: * multiple experts receive meaningful responsibility mass; * assigned utility > unassigned utility for at least several experts; * residual correlation falls materially below the Suite7/8 ~0.99 regime; * oracle-responsibility performance is meaningfully above Suite7; * learned router captures a substantial fraction of that oracle gain; * final AP/F2 gains survive temporal validation without relying on holdout labels. If oracle improves strongly but learned router does not, the next experiment should focus on gate predictability / environment-aware routing. If oracle itself remains weak, further router work is not justified; the expert responsibility definition/objective must change (e.g. failure-boundary/ranking-aware utility rather than BCE utility).