SUITE6 — TARGET-ONLY RESIDUAL MoE ABLATION =========================================== Research question ----------------- Does the Suite5 cross-faculty Residual-MoE suffer negative transfer on large/ well-supported faculties, while cross-faculty sharing helps data-poor faculties? This suite trains the exact same residual-MoE architecture independently on four faculties and compares it to the exact frozen Committee and exact Suite5 shared Residual-MoE scores: 1. khoa_cong_nghe_thong_tin 2. khoa_quan_tri_kinh_doanh_va_du_lich 3. khoa_tieng_duc 4. khoa_quoc_te_hoc The Suite5 canonical cache is REUSED. No RF / ExtraTrees / HGB Committee models are retrained. The completed Suite5 shared Residual-MoE is also NOT retrained. Its target scores and feature schema are read from --shared_results_dir. FAIRNESS CONSTRAINTS -------------------- The target-only model deliberately uses: - the same frozen faculty-specific Committee; - the same OOF historical Committee predictions; - the same 4 residual experts; - the same residual bound / losses / optimizer / validation blocks; - the exact common raw-feature list saved by the prior Suite5 shared run; - the same 13 Committee meta features + 5 support features; - the same Suite5 target-only seed convention. The intended changed factor is the residual-training framework: Suite5: all faculties jointly, macro-faculty epoch selection. Suite6: one faculty at a time, that faculty's chronological validation selects epoch. Thus Suite6 is a framework-level ablation (shared vs independent), not a claim that only one internal variable changed. REQUIRED INPUTS --------------- 1. Completed Suite5 canonical cache, e.g. /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache 2. Completed Suite5 shared result directory, e.g. /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe The shared result directory must contain for every fold: fold_N/feature_schema.json fold_N/_scores.csv The score files are alignment-checked against Suite5 cache labels and Committee scores before they are accepted. PRIMARY RUN ----------- From the suite6 directory: mkdir -p /root/disk_3tb/earlywarning/30072026/results/suite6_target_only_residual mkdir -p /root/disk_3tb/earlywarning/30072026/log python3 train_target_only_residual_ablation.py \ --cache_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \ --shared_results_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe \ --models_dir /root/disk_3tb/earlywarning/30072026/results/suite6_target_only_residual \ --faculties khoa_cong_nghe_thong_tin,khoa_quan_tri_kinh_doanh_va_du_lich,khoa_tieng_duc,khoa_quoc_te_hoc \ --dev_folds 1,2,3,4,5 \ --holdout_folds 6,7 \ --seed 42 \ --device auto \ --n_experts 4 \ --faculty_dim 8 \ --encoder_hidden 256 \ --representation_dim 128 \ --expert_hidden 64 \ --delta_max 2.0 \ --dropout 0.10 \ --router_temperature 1.0 \ --epochs 60 \ --min_epochs 8 \ --patience 8 \ --fallback_epochs 20 \ --val_blocks 2 \ --min_train_rows 800 \ --min_val_rows 200 \ --min_target_only_failures 20 \ --batch_size 1024 \ --predict_batch_size 4096 \ --lr 0.001 \ --weight_decay 0.0001 \ --grad_clip 5.0 \ --print_every 5 \ --router_lambda 0.5 \ --specialization_lambda 0.01 \ --magnitude_lambda 0.01 \ --utility_temperature 0.25 \ --sampling_beta 0.9999 \ --macro_f2_weight 1.0 \ --min_selection_improvement 0.00001 \ --threshold_beta 2.0 \ --min_precision 0.15 \ --max_warning_ratio 2.5 \ --min_flags 5 \ > /root/disk_3tb/earlywarning/30072026/log/suite6_target_only_residual.log 2>&1 Monitor: tail -f /root/disk_3tb/earlywarning/30072026/log/suite6_target_only_residual.log The shorter command below is equivalent because the explicit values above are current defaults: python3 train_target_only_residual_ablation.py \ --cache_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \ --shared_results_dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe \ --models_dir /root/disk_3tb/earlywarning/30072026/results/suite6_target_only_residual \ > /root/disk_3tb/earlywarning/30072026/log/suite6_target_only_residual.log 2>&1 OPTIONAL: ADD ORIGINAL V2 / SUITE4 COUNCIL REFERENCES ----------------------------------------------------- Suite6 already compares Committee vs Suite5 Shared vs Suite6 Target-only using exact row-level scores. To append historical V2/Council metrics to the final CSV, create a reference CSV with collect_reference_metrics.py. Example Council references (adjust log paths if yours differ): python3 collect_reference_metrics.py \ --logs \ '/root/disk_3tb/earlywarning/30072026/log/council_khoa_cong_nghe_thong_tin.log' \ '/root/disk_3tb/earlywarning/30072026/log/council_khoa_quan_tri_kinh_doanh_va_du_lich.log' \ '/root/disk_3tb/earlywarning/30072026/log/council_khoa_tieng_duc.log' \ '/root/disk_3tb/earlywarning/30072026/log/council_khoa_quoc_te_hoc.log' \ --default_model competence_council \ --models competence_council \ --output /root/disk_3tb/earlywarning/30072026/results/suite6_council_reference.csv Example original IT V2 reference: python3 collect_reference_metrics.py \ --logs '/root/disk_3tb/earlywarning/30072026/log/final_train_v2.log' \ --faculty khoa_cong_nghe_thong_tin \ --default_model v2 \ --models v2 \ --output /root/disk_3tb/earlywarning/30072026/results/suite6_v2_reference.csv Combine them: python3 - <<'PY' import pandas as pd paths = [ '/root/disk_3tb/earlywarning/30072026/results/suite6_council_reference.csv', '/root/disk_3tb/earlywarning/30072026/results/suite6_v2_reference.csv', ] pd.concat([pd.read_csv(p) for p in paths], ignore_index=True).to_csv( '/root/disk_3tb/earlywarning/30072026/results/suite6_reference_metrics.csv', index=False, ) PY Then add to the main command: --reference_metrics /root/disk_3tb/earlywarning/30072026/results/suite6_reference_metrics.csv MAIN OUTPUTS ------------ suite6_fold_metrics.csv Every fold × faculty × model, AP/AUC and F2 at 10/15/20/25% budgets. suite6_holdout_pooled.csv Pooled Fold6+7 ranking and equal-budget metrics for: committee residual_moe_shared residual_moe_target_only suite6_warning_metrics.csv Fixed-threshold and drift-aware holdout metrics / confusion matrices in CSV form. suite6_threshold_policies.csv Each model's development-selected threshold and warning ratio. Policies are independently tuned on identical development folds and frozen before holdout. suite6_local_vs_shared_transfer_diagnostic.csv Central diagnostic table. Shows local-vs-shared AP, F2@20 and drift-F2 deltas. evidence_pattern is descriptive only: LOCAL_STRONGER = at least 2/3 of AP, F2@20, drift-F2 favor local SHARED_STRONGER = at least 2/3 favor shared MIXED = otherwise suite6_training_summary.csv Historical rows/failures, validation rows/failures, local selected epoch, previous shared selected epoch when checkpoint is available, local/shared NULL weight, entropy, max weight, correction magnitude and residual correlation. suite6_target_only_specialization.csv Target-only gate/residual behavior by all/failure/normal/history regime. suite6_shared_vs_local_router_diagnostic.csv Holdout-average comparison of gate entropy, NULL usage, correction magnitude, selected epoch and residual correlation between shared and local systems. suite6_official_comparison.csv Central final table, optionally augmented with V2/Council references. Per fold/faculty: _suite6_scores.csv _target_only_training_history.csv _target_only_best_validation.csv _target_only_model.pt _target_only_scaler.npz _target_only_specialization.csv _target_only_residual_correlations.csv HOW TO INTERPRET THIS EXPERIMENT -------------------------------- The strongest evidence for a shared-private next architecture would be: IT / Business: target-only > shared German / International Studies: shared >= target-only especially when the same direction appears in both ranking (AP / F2@20) and the frozen operational policies (drift F2), not just one metric. Do not infer negative transfer from AP alone. Inspect: - AP and AUC; - F2@10/15/20/25; - fixed and drift F2; - recall / warning rate; - selected epoch; - NULL weight; - gate entropy and max weight; - residual expert pairwise correlations; - correction magnitude on failures vs normals. If target-only wins only because it selects a very different operating threshold while AP/AUC stay equal, that is not strong evidence of representation-level negative transfer. If target-only improves AP/AUC and F2 across budgets, the negative-transfer interpretation is much stronger.