SUITE10 — SEMANTIC ACADEMIC-MECHANISM SPECIALISTS ================================================== PURPOSE ------- Suite10 tests the next responsibility hypothesis after Suite9: Suite9 proved that different training responsibilities can break residual-expert collapse even when the experts see almost identical views. However, Suite9's responsibility was label-conditioned counterfactual BCE utility, which is not a stable pre-outcome domain and was difficult for the router to predict. Suite10 keeps the successful principle (different experts receive different responsibilities) but replaces label-conditioned ownership with four observable, pre-outcome academic mechanisms: E1 student_sparse weak / absent student academic history; exact new-student and missing-history evidence receives strongest membership. E2 course_sparse weak / absent prior course support; exact new-course and missing-course-history evidence receives strongest membership. E3 temporal_deterioration recent academic state is worse than longer-term state. Signals include recent-vs-historical semester GPA, course grade, component grade, latest-vs-cumulative GPA, warning-rate increase, and conduct decline. E4 history_relational substantial student history plus enough course / same-course relational evidence to justify student-course-specific reasoning. E0 NULL / Committee no validated specialist mechanism receives sufficient authority. IMPORTANT: labels never determine domain membership or target routing. Committee uncertainty is NOT an expert domain. It only controls how much total specialist authority is permitted. CONTROLLED BACKBONE ------------------- Suite10 reuses: * Suite5 canonical RF / ExtraTrees / HGB Committee cache; * exact Suite5 common 136-feature schema; * Suite7 shared cross-faculty encoder; * Suite7 faculty-specific adapters; * Suite7 residual-expert architecture; * chronological OOF history and development/holdout protocol. No Committee cache is regenerated. No neural gate is trained in Suite10. CHRONOLOGY ---------- For each fold, the historical OOF data are divided into: earlier history -> specialist training older validation block -> expert epoch selection + reliability estimation newest validation block-> whole-system deploy/fallback decision target fold -> untouched evaluation If a faculty has only one usable validation block, reliability and selection must share it; this is explicitly printed as independent_rel_selection=False. For final target refit, Suite10 loads the actual fitted Suite7 checkpoint, fits semantic calibrators on all historical OOF rows, trains each enabled expert for its chronology-selected number of epochs, then routes the unseen target. DOMAIN RESPONSIBILITY --------------------- Each row receives soft m_k(x) in [0,1]. Domains may overlap. Training uses m_k^domain_power (default power=2) so core rows have much stronger influence than fringe rows. Each expert BCE is normalized by its own domain mass. Core-domain reporting defaults to: m_k >= 0.60 This threshold is for evaluation / reliability; the training responsibility is soft rather than a hard bucket. EXPERT CONTRACT --------------- Each expert is validated inside its intended future domain against the frozen Committee anchor. Suite10 reports separately: in-domain Committee AP/AUC/F2@20 in-domain expert AP/AUC/F2@20 out-of-domain Committee / expert metrics specialization contrast: in-domain uplift - out-of-domain uplift A specialist is disabled if no post-training epoch achieves positive enough in-domain selection. Epoch 0 cannot enable a semantic specialist; this prevents arbitrary pre-existing Suite7 expert slots from being relabeled as meaningful mechanisms without semantic training. RELIABILITY ----------- For each faculty/expert, reliability is estimated only on held-out historical reliability blocks from in-domain AP + F2@20 uplift. It is shrunk toward a global cross-faculty reliability estimate. Shrinkage support is constrained by both rows AND failures: effective_support = min(domain_rows, domain_failures * reliability_failure_equiv_rows) This prevents a large low-prevalence domain from becoming overconfident because it contains many rows but very few failure examples. STRUCTURED ROUTING ------------------ Expert composition is determined by: base_ik = domain_score_ik * reliability_fk Uncertainty never changes relative expert composition. It controls only total specialist authority: cap_i = authority_min + (authority_max - authority_min) * uncertainty_i authority_i = domain_strength_i * reliability_strength_i * cap_i w_null = 1 - authority_i The default authority cap range is 0.10 to 0.70. PRIMARY DIAGNOSTICS ------------------- Suite10 writes: suite10_domain_viability.csv Per fold/faculty/domain: core rows, failures, prevalence, mean membership, Committee AP/AUC/F2@20 in the domain. suite10_domain_overlap.csv Pairwise domain intersections and Jaccard overlap. suite10_domain_drift.csv Historical/recent/target domain score and core-rate changes. suite10_expert_contracts.csv In-domain vs out-of-domain expert uplift and specialization contrast. suite10_reliability.csv Local raw, global, and shrunk faculty-domain reliability. suite10_authority_diagnostics.csv NULL rate, authority, uncertainty, multi-domain rate, residual correlation. suite10_residual_correlations.csv Residual-expert functional diversity. suite10_fold_metrics.csv AP/AUC/F2@10/15/20/25 for Committee, Shared, Suite7, optional Suite8/Suite9, Suite10 candidate and Suite10 safety-final. suite10_holdout_mean_fold.csv PRIMARY ranking comparison. Metrics are computed within each target semester and then averaged, avoiding raw-score-scale mixing between folds. suite10_holdout_pooled_raw.csv Raw pooled score comparison retained mainly for calibration/score-semantic diagnosis; do not interpret it alone if fallback/model regimes differ. suite10_warning_metrics.csv Development-frozen fixed and drift-aware operational metrics. suite10_comparison.csv Compact faculty-level Suite7-vs-Suite10 comparison. Each fold also stores: semantic_specialist_model.pt semantic_scaler.npz semantic_domain_calibrator.json expert_training_history.csv deployment_selection_by_faculty.csv domain_viability.csv / domain_overlap.csv / expert_contracts.csv / reliability.csv / authority_diagnostics.csv / domain_drift.csv _scores.csv Per-row score files contain domain scores, uncertainty, authority, NULL/expert weights, residuals, counterfactual expert probabilities, Committee, earlier model scores when supplied, Suite10 candidate, and Suite10 final. RECOMMENDED SERVER RUN ---------------------- Assuming suite10 is copied to: /root/disk_3tb/earlywarning/30072026/suite10 run: cd /root/disk_3tb/earlywarning/30072026/suite10 mkdir -p /root/disk_3tb/earlywarning/30072026/results/suite10_semantic_specialists mkdir -p /root/disk_3tb/earlywarning/30072026/log python3 train_suite10_semantic_specialists.py \ --cache_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \ --shared_results_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe \ --suite7_results_dir /root/disk_3tb/earlywarning/30072026/results/suite7_shared_private_adapter \ --suite8_results_dir /root/disk_3tb/earlywarning/30072026/results/suite8_true_specialist \ --suite9_results_dir /root/disk_3tb/earlywarning/30072026/results/suite9_competitive_responsibility \ --models_dir /root/disk_3tb/earlywarning/30072026/results/suite10_semantic_specialists \ --faculties all \ --dev_folds 1,2,3,4,5 \ --holdout_folds 6,7 \ --seed 42 \ --device auto \ --epochs 10 \ --min_epochs 3 \ --patience 4 \ --expert_lr 0.0005 \ --domain_power 2.0 \ --domain_train_floor 0.05 \ --domain_route_floor 0.10 \ --domain_core_threshold 0.60 \ --domain_min_rows 50 \ --domain_min_failures 5 \ --reliability_gain 0.03 \ --reliability_shrink_rows 200 \ --reliability_failure_equiv_rows 20 \ --authority_min 0.10 \ --authority_max 0.70 \ --val_blocks 2 \ --batch_size 1024 \ --predict_batch_size 4096 \ --sampling_beta 0.9999 \ --magnitude_lambda 0.01 \ --macro_f2_weight 1.0 \ --min_selection_improvement 0.001 \ --threshold_beta 2.0 \ --min_precision 0.15 \ --max_warning_ratio 2.5 \ --min_flags 5 \ --verbose_operational \ > /root/disk_3tb/earlywarning/30072026/log/suite10_semantic_specialists.log 2>&1 Monitor: tail -f /root/disk_3tb/earlywarning/30072026/log/suite10_semantic_specialists.log I recommend the first run exactly with these defaults. Do not tune the domain thresholds or authority values before seeing the viability / overlap / contract reports; otherwise the first experiment stops being a clean test of the semantic responsibility hypothesis. OPTIONAL V2 / COUNCIL REFERENCES -------------------------------- `collect_reference_metrics.py` is included. If you already have its reference CSV, pass: --reference_metrics /path/to/reference_metrics.csv and Suite10 also writes `suite10_official_comparison.csv`. SUCCESS / FAILURE INTERPRETATION -------------------------------- Strong evidence for semantic specialization: * multiple experts have positive held-out in-domain uplift; * in-domain uplift > out-of-domain uplift; * residual correlation falls meaningfully from Suite7's ~0.99; * domain core rates remain reasonably stable through time; * reliability is nontrivial and not concentrated in only one faculty; * candidate mean-fold AP/F2 improves without requiring label-conditioned routing. Important failure modes: * domain has too few failures -> domain definition is not trainable; * expert improves globally but not in its intended domain -> semantic assignment wrong; * in-domain and out-domain uplift equal -> expert is not truly specialized; * domain drifts strongly in target semester -> mechanism definition unstable; * reliability shrinks near zero -> historical evidence does not support that specialist; * residual correlation stays ~0.99 -> semantic responsibilities are too overlapping/weak; * candidate improves but final fixed metrics collapse -> score semantics/calibration is the next problem, not domain responsibility itself.