SUITE8 — TRUE SPECIALIST DISCOVERY =================================== Research question ----------------- Suite5/6/7 consistently show residual-expert correlations around 0.99 and router entropy near its maximum. Does forcing residual experts to consume distinct learned latent views create genuine complementary specialists and make routing useful? Controlled baseline ------------------- Suite8 starts from Suite7. It preserves: - faculty-specific frozen RF/ET/HGB Committee cache; - exact 136-feature common raw schema + Committee meta + support features; - faculty embedding and shared encoder; - 128->16->128 faculty-private bottleneck adapters; - NULL/Committee route; - residual expert architecture and bounded logit corrections; - capability-based router supervision; - chronological macro-faculty validation; - equal-faculty/effective-number sampling; - development-frozen fixed and drift-aware operational evaluation. Only expert input specialization changes. Learned latent ownership ------------------------ For K experts and D adapted latent dimensions, learn ownership logits Omega[K,D]. Each latent dimension is normalized ACROSS experts: alpha[:,d] = softmax(Omega[:,d] / T_view) and each expert receives a scaled view scale[k,d] = (1-s) + s*K*alpha[k,d] h_k = h_faculty * scale[k] When alpha is uniform, scale=1 exactly and every h_k equals the Suite7 adapted representation. Small random ownership-logit initialization breaks symmetry while preserving a near-Suite7 starting point. Residual output layers remain zero, so the complete model still starts exactly at the Committee score. Two mild ownership regularizers are added: 1. minimize normalized per-dimension ownership entropy (encourage distinct owners), 2. minimize expert load imbalance (prevent one expert owning all dimensions). IMPORTANT: there is deliberately NO direct residual-output decorrelation penalty in Suite8. If feature ownership diversifies but residual correlations remain near 1, that is evidence that collapse comes from the downstream residual/competence objective rather than identical inputs. Runtime / cache --------------- NO new Committee cache is required. Reuse: /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache Suite5 and Suite7 score files are read-only references. They are not retrained: Suite5: /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe Suite7: /root/disk_3tb/earlywarning/30072026/results/suite7_shared_private_adapter Only seven new Suite8 models (one per target fold) are trained. Recommended run --------------- cd /root/disk_3tb/earlywarning/30072026/suite8 mkdir -p /root/disk_3tb/earlywarning/30072026/results/suite8_true_specialist mkdir -p /root/disk_3tb/earlywarning/30072026/log python3 train_suite8_true_specialist.py \ --cache_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \ --shared_results_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe \ --suite7_results_dir /root/disk_3tb/earlywarning/30072026/results/suite7_shared_private_adapter \ --models_dir /root/disk_3tb/earlywarning/30072026/results/suite8_true_specialist \ --faculties all \ --dev_folds 1,2,3,4,5 \ --holdout_folds 6,7 \ --seed 42 \ --device auto \ --n_experts 4 \ --faculty_dim 8 \ --encoder_hidden 256 \ --representation_dim 128 \ --expert_hidden 64 \ --delta_max 2.0 \ --dropout 0.10 \ --router_temperature 1.0 \ --adapter_dim 16 \ --adapter_dropout 0.0 \ --adapter_scale 1.0 \ --view_temperature 1.0 \ --view_strength 0.50 \ --view_init_std 0.01 \ --view_entropy_lambda 0.02 \ --view_balance_lambda 0.02 \ --top_owned_dims 8 \ --epochs 60 \ --min_epochs 8 \ --patience 8 \ --fallback_epochs 20 \ --val_blocks 2 \ --min_train_rows 800 \ --min_val_rows 200 \ --batch_size 1024 \ --predict_batch_size 4096 \ --lr 0.001 \ --weight_decay 0.0001 \ --grad_clip 5.0 \ --print_every 5 \ --router_lambda 0.5 \ --specialization_lambda 0.01 \ --magnitude_lambda 0.01 \ --utility_temperature 0.25 \ --sampling_beta 0.9999 \ --macro_f2_weight 1.0 \ --min_selection_improvement 0.00001 \ --threshold_beta 2.0 \ --min_precision 0.15 \ --max_warning_ratio 2.5 \ --min_flags 5 \ > /root/disk_3tb/earlywarning/30072026/log/suite8_true_specialist.log 2>&1 Monitor: tail -f /root/disk_3tb/earlywarning/30072026/log/suite8_true_specialist.log Do not tune the view hyperparameters before the first full run. The purpose of this run is to test the architectural hypothesis cleanly. Main outputs ------------ suite8_fold_metrics.csv Per fold/faculty Committee, Suite5 Shared, Suite7 Adapter, Suite8 Specialist AP/AUC and F2 at 10/15/20/25% budgets. suite8_holdout_pooled.csv Pooled Fold6+7 ranking/equal-budget metrics for all four models. suite8_warning_metrics.csv suite8_threshold_policies.csv Independently development-tuned/frozen fixed and drift-aware policies for all four models, including precision/recall/F1/F2/rate and confusion matrices. suite8_true_specialist_comparison.csv Main Suite8-vs-Suite7 causal table. Includes AP/AUC/F2@20/fixed-F2/drift-F2 deltas and SPECIALIST_STRONGER / ADAPTER_STRONGER / MIXED label. suite8_specialist_diagnostics.csv Faculty/fold/group diagnostics for all/failures/normals/history regimes: NULL weight, router entropy, max weight, residual stats/correlation, adapter behavior, ownership entropy/cosine, view cosine, and counterfactual utilities. suite8_expert_ownership.csv Full learned ownership matrix: fold, expert, latent dimension, ownership, scale, and rank within expert. suite8_counterfactual_utility.csv NULL/expert counterfactual win rates and mean utility overall/failure/normal. suite8_residual_correlations.csv Pairwise residual-output correlations by faculty/fold. suite8_collapse_summary.csv Main mechanistic comparison on holdout: Suite7 vs Suite8 residual correlation and router entropy, plus Suite8 ownership/view diversity and performance deltas. suite8_training_summary.csv Selected epoch and final fold-level ownership/view/residual diagnostics. Per fold: fold_N/true_specialist_model.pt fold_N/true_specialist_scaler.npz fold_N/true_specialist_training_history.csv fold_N/true_specialist_best_validation_by_faculty.csv fold_N/expert_ownership.csv fold_N/counterfactual_utility.csv fold_N/specialist_diagnostics.csv fold_N/residual_correlations.csv fold_N/_scores.csv Optional V2 / Council references -------------------------------- Pass the same combined reference CSV used in Suite6/7: --reference_metrics /path/to/v2_council_reference.csv Suite8 then writes suite8_official_comparison.csv. Interpretation hierarchy ------------------------ 1. Ownership entropy/cosine remain near uniform/high: feature-view specialization itself failed to emerge. 2. Ownership and actual view cosine diversify, but residual correlation remains ~.99: experts see different information but the residual/capability objective drives them back to the same correction. This strongly motivates Experiment D/objective redesign. 3. Residual correlation falls but router entropy remains near 1: specialists exist, but the gate cannot distinguish competence reliably. 4. Residual correlation falls AND router entropy/max-weight differentiate AND AP/F2 improve: genuine specialist discovery is working. 5. Diversity rises while AP/F2 worsen: diversity is being manufactured rather than discovering useful mechanisms; do not increase diversity regularization blindly. Primary success criterion ------------------------- Mechanistic success requires BOTH: - meaningful reduction from Suite7 residual correlation (~0.99), and - stable/improved ranking and F2 across faculties. Performance improvement without reduced expert collapse means Suite8 did not answer the specialization hypothesis. Reduced correlation without performance improvement is also not enough to call the architecture successful.