[ENV] python=3.13.5 torch=2.7.1+cu118 device=cuda [ENV] faculties=14 folds=[1, 2, 3, 4, 5, 6, 7] dev=[1, 2, 3, 4, 5] holdout=[6, 7] [SCIENTIFIC CONTRACT] OOF Committee is the fixed strong anchor; Suite21 only learns bounded residual evidence; alpha=0 exactly recovers Committee; target labels never select epochs/authority [TRAINING CHANGE] faculty-balanced sampling only; NO class-rebalancing sampler, to avoid Suite20 prior distortion ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 1 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=240,672 failures=4,445 (1.85%) target=45,800 failures=1,083 (2.36%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 5222 0.0362 0.0000 4.9111 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 5672 0.0386 0.0000 4.8900 6.0000 khoa_quoc_te_hoc 2363 0.0131 0.0000 4.8316 6.0000 khoa_tieng_anh 8164 0.0181 0.0000 3.9103 4.0000 khoa_tieng_bo_dao_nha 768 0.0391 0.0000 4.0026 4.0000 khoa_tieng_duc 1900 0.0242 0.0000 4.0416 4.0000 khoa_tieng_han_quoc 2904 0.0069 0.0000 4.2073 4.0000 khoa_tieng_italia 2166 0.0512 0.0000 4.1898 4.0000 khoa_tieng_nga 1588 0.0642 0.0000 3.9433 4.0000 khoa_tieng_nhat 2840 0.0060 0.0000 4.2380 4.0000 khoa_tieng_phap 2606 0.0161 0.0000 4.0177 4.0000 khoa_tieng_tay_ban_nha 1401 0.0450 0.0000 4.0942 4.0000 khoa_tieng_trung_quoc 7393 0.0047 0.0000 4.6633 5.0000 khoa_viet_nam_hoc 813 0.0369 0.0000 4.9225 6.0000 [SPLIT] train=141,668 validation=99,004 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11793 best_alpha=0.25 robust=+0.00093 meanΔAP=+0.0014 medΔAP=+0.0022 q25=-0.0022 wins=9/14 [SELECT] epoch=02/10 loss=0.10363 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=03/10 loss=0.09898 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=04/10 loss=0.09442 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=05/10 loss=0.09269 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=06/10 loss=0.09105 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [VARIANT FLAT_RESIDUAL] selected_epochs=1 selected_alpha=0.25 robust=+0.000927 [REFIT] epoch=01/1 loss=0.11288 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.14054 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=02/10 loss=0.12373 best_alpha=0.25 robust=+0.00047 meanΔAP=+0.0006 medΔAP=+0.0014 q25=-0.0028 wins=8/14 [SELECT] epoch=03/10 loss=0.11607 best_alpha=0.25 robust=+0.00155 meanΔAP=+0.0020 medΔAP=+0.0034 q25=-0.0024 wins=9/14 [SELECT] epoch=04/10 loss=0.11145 best_alpha=0.25 robust=+0.00054 meanΔAP=+0.0019 medΔAP=-0.0003 q25=-0.0019 wins=7/14 [SELECT] epoch=05/10 loss=0.10783 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=06/10 loss=0.10558 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=3 selected_alpha=0.25 robust=+0.001548 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/3 loss=0.13372 [REFIT] epoch=03/3 loss=0.11703 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.14184 best_alpha=0.50 robust=+0.00367 meanΔAP=+0.0050 medΔAP=+0.0039 q25=-0.0011 wins=10/14 [SELECT] epoch=02/10 loss=0.12446 best_alpha=0.50 robust=+0.00254 meanΔAP=+0.0051 medΔAP=+0.0023 q25=-0.0033 wins=7/14 [SELECT] epoch=03/10 loss=0.11811 best_alpha=0.25 robust=+0.00123 meanΔAP=+0.0018 medΔAP=+0.0022 q25=-0.0023 wins=9/14 [SELECT] epoch=04/10 loss=0.11366 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=05/10 loss=0.11148 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [SELECT] epoch=06/10 loss=0.10938 best_alpha=0.00 robust=+0.00000 meanΔAP=+0.0000 medΔAP=+0.0000 q25=+0.0000 wins=3/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=0.50 robust=+0.003669 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.13774 SUITE21 TARGET FOLD 1 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 5222 0.0362 0.2032 0.7519 0.0321 0.1197 0.6614 0.3472 khoa_cong_nghe_thong_tin suite21_flat_residual 5222 0.0362 0.2170 0.7606 0.0317 0.1245 0.6878 0.3611 khoa_cong_nghe_thong_tin suite21_temporal_residual 5222 0.0362 0.2136 0.7595 0.0318 0.1216 0.6720 0.3528 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 5222 0.0362 0.2307 0.7608 0.0315 0.1216 0.6720 0.3528 khoa_quan_tri_kinh_doanh_va_du_lich committee 5672 0.0386 0.5174 0.9168 0.0281 0.1684 0.8721 0.4751 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 5672 0.0386 0.5223 0.9186 0.0273 0.1693 0.8767 0.4776 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 5672 0.0386 0.5207 0.9190 0.0276 0.1693 0.8767 0.4776 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 5672 0.0386 0.5257 0.9200 0.0271 0.1693 0.8767 0.4776 khoa_quoc_te_hoc committee 2363 0.0131 0.0704 0.8048 0.0127 0.0465 0.7097 0.1843 khoa_quoc_te_hoc suite21_flat_residual 2363 0.0131 0.0813 0.8096 0.0126 0.0486 0.7419 0.1926 khoa_quoc_te_hoc suite21_temporal_residual 2363 0.0131 0.0745 0.8047 0.0127 0.0486 0.7419 0.1926 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2363 0.0131 0.0770 0.8057 0.0127 0.0444 0.6774 0.1759 khoa_tieng_anh committee 8164 0.0181 0.5856 0.9353 0.0108 0.0833 0.9189 0.3056 khoa_tieng_anh suite21_flat_residual 8164 0.0181 0.5909 0.9362 0.0108 0.0833 0.9189 0.3056 khoa_tieng_anh suite21_temporal_residual 8164 0.0181 0.5918 0.9344 0.0110 0.0827 0.9122 0.3034 khoa_tieng_anh suite21_guarded_temporal_residual 8164 0.0181 0.5971 0.9348 0.0109 0.0802 0.8851 0.2944 khoa_tieng_bo_dao_nha committee 768 0.0391 0.3146 0.8200 0.0336 0.1104 0.5667 0.3102 khoa_tieng_bo_dao_nha suite21_flat_residual 768 0.0391 0.3214 0.8279 0.0333 0.1104 0.5667 0.3102 khoa_tieng_bo_dao_nha suite21_temporal_residual 768 0.0391 0.3146 0.8240 0.0333 0.1234 0.6333 0.3467 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 768 0.0391 0.2664 0.8359 0.0335 0.1169 0.6000 0.3285 khoa_tieng_duc committee 1900 0.0242 0.2622 0.8663 0.0212 0.0921 0.7609 0.3103 khoa_tieng_duc suite21_flat_residual 1900 0.0242 0.2635 0.8723 0.0209 0.0921 0.7609 0.3103 khoa_tieng_duc suite21_temporal_residual 1900 0.0242 0.2652 0.8634 0.0210 0.0921 0.7609 0.3103 khoa_tieng_duc suite21_guarded_temporal_residual 1900 0.0242 0.2608 0.8836 0.0210 0.0974 0.8043 0.3280 khoa_tieng_han_quoc committee 2904 0.0069 0.0544 0.8479 0.0067 0.0275 0.8000 0.1210 khoa_tieng_han_quoc suite21_flat_residual 2904 0.0069 0.0533 0.8490 0.0068 0.0258 0.7500 0.1135 khoa_tieng_han_quoc suite21_temporal_residual 2904 0.0069 0.0541 0.8493 0.0068 0.0275 0.8000 0.1210 khoa_tieng_han_quoc suite21_guarded_temporal_residual 2904 0.0069 0.0556 0.8505 0.0067 0.0258 0.7500 0.1135 khoa_tieng_italia committee 2166 0.0512 0.2598 0.7668 0.0457 0.1386 0.5405 0.3421 khoa_tieng_italia suite21_flat_residual 2166 0.0512 0.2711 0.7841 0.0449 0.1478 0.5766 0.3649 khoa_tieng_italia suite21_temporal_residual 2166 0.0512 0.2953 0.7818 0.0449 0.1455 0.5676 0.3592 khoa_tieng_italia suite21_guarded_temporal_residual 2166 0.0512 0.2869 0.7810 0.0443 0.1432 0.5586 0.3535 khoa_tieng_nga committee 1588 0.0642 0.3476 0.8074 0.0532 0.1887 0.5882 0.4132 khoa_tieng_nga suite21_flat_residual 1588 0.0642 0.3502 0.8184 0.0523 0.1824 0.5686 0.3994 khoa_tieng_nga suite21_temporal_residual 1588 0.0642 0.3520 0.8090 0.0526 0.1855 0.5784 0.4063 khoa_tieng_nga suite21_guarded_temporal_residual 1588 0.0642 0.3545 0.8303 0.0522 0.2013 0.6275 0.4408 khoa_tieng_nhat committee 2840 0.0060 0.5024 0.9301 0.0054 0.0246 0.8235 0.1101 khoa_tieng_nhat suite21_flat_residual 2840 0.0060 0.5009 0.9305 0.0052 0.0246 0.8235 0.1101 khoa_tieng_nhat suite21_temporal_residual 2840 0.0060 0.4812 0.9357 0.0053 0.0246 0.8235 0.1101 khoa_tieng_nhat suite21_guarded_temporal_residual 2840 0.0060 0.4770 0.9381 0.0051 0.0246 0.8235 0.1101 khoa_tieng_phap committee 2606 0.0161 0.2279 0.8058 0.0144 0.0461 0.5714 0.1742 khoa_tieng_phap suite21_flat_residual 2606 0.0161 0.2500 0.8216 0.0142 0.0557 0.6905 0.2104 khoa_tieng_phap suite21_temporal_residual 2606 0.0161 0.2374 0.8156 0.0143 0.0518 0.6429 0.1959 khoa_tieng_phap suite21_guarded_temporal_residual 2606 0.0161 0.2280 0.8258 0.0141 0.0518 0.6429 0.1959 khoa_tieng_tay_ban_nha committee 1401 0.0450 0.1689 0.7736 0.0423 0.1179 0.5238 0.3102 khoa_tieng_tay_ban_nha suite21_flat_residual 1401 0.0450 0.1973 0.7896 0.0414 0.1214 0.5397 0.3195 khoa_tieng_tay_ban_nha suite21_temporal_residual 1401 0.0450 0.1962 0.7795 0.0417 0.1214 0.5397 0.3195 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1401 0.0450 0.1851 0.7912 0.0422 0.1214 0.5397 0.3195 khoa_tieng_trung_quoc committee 7393 0.0047 0.1112 0.7903 0.0045 0.0169 0.7143 0.0772 khoa_tieng_trung_quoc suite21_flat_residual 7393 0.0047 0.1159 0.7890 0.0045 0.0169 0.7143 0.0772 khoa_tieng_trung_quoc suite21_temporal_residual 7393 0.0047 0.1186 0.7891 0.0045 0.0169 0.7143 0.0772 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 7393 0.0047 0.1219 0.7881 0.0045 0.0169 0.7143 0.0772 khoa_viet_nam_hoc committee 813 0.0369 0.1373 0.7654 0.0399 0.0982 0.5333 0.2827 khoa_viet_nam_hoc suite21_flat_residual 813 0.0369 0.1279 0.7672 0.0427 0.1043 0.5667 0.3004 khoa_viet_nam_hoc suite21_temporal_residual 813 0.0369 0.1360 0.7661 0.0436 0.0920 0.5000 0.2650 khoa_viet_nam_hoc suite21_guarded_temporal_residual 813 0.0369 0.1162 0.7899 0.0433 0.0920 0.5000 0.2650 SUITE21 TARGET FOLD 1 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.2032 0.2307 0.0275 0.0046 0.9979 0.9617 0.0094 -0.0030 khoa_quan_tri_kinh_doanh_va_du_lich 0.5174 0.5257 0.0083 0.0029 0.9985 0.9700 0.0268 0.0009 khoa_quoc_te_hoc 0.0704 0.0770 0.0066 0.0030 0.9764 0.8689 0.0121 0.0016 khoa_tieng_anh 0.5856 0.5971 0.0115 0.0021 0.9970 0.9798 0.0142 0.0005 khoa_tieng_bo_dao_nha 0.3146 0.2664 -0.0482 0.0077 0.9785 0.8831 0.0200 0.0050 khoa_tieng_duc 0.2622 0.2608 -0.0014 0.0039 0.9846 0.8789 0.0182 0.0014 khoa_tieng_han_quoc 0.0544 0.0556 0.0012 0.0003 0.9942 0.9673 -0.0006 -0.0003 khoa_tieng_italia 0.2598 0.2869 0.0271 0.0067 0.9913 0.9053 0.0238 0.0015 khoa_tieng_nga 0.3476 0.3545 0.0069 0.0078 0.9581 0.8648 0.0183 0.0018 khoa_tieng_nhat 0.5024 0.4770 -0.0255 0.0020 0.9970 0.9806 0.0205 -0.0016 khoa_tieng_phap 0.2279 0.2280 0.0001 0.0029 0.9934 0.9290 0.0273 -0.0002 khoa_tieng_tay_ban_nha 0.1689 0.1851 0.0162 0.0043 0.9847 0.8893 0.0007 -0.0024 khoa_tieng_trung_quoc 0.1112 0.1219 0.0107 0.0008 0.9957 0.9601 -0.0011 -0.0005 khoa_viet_nam_hoc 0.1373 0.1162 -0.0211 0.0116 0.9845 0.9141 0.0236 0.0067 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 2 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=286,472 failures=5,528 (1.93%) target=42,122 failures=933 (2.21%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 4598 0.0331 0.0000 5.2275 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 5530 0.0300 0.0000 5.2047 6.0000 khoa_quoc_te_hoc 2095 0.0091 0.0000 5.1484 6.0000 khoa_tieng_anh 8328 0.0146 0.0000 4.5702 5.0000 khoa_tieng_bo_dao_nha 716 0.0419 0.0000 4.6411 5.0000 khoa_tieng_duc 1893 0.0280 0.0000 4.8193 5.0000 khoa_tieng_han_quoc 3337 0.0087 0.0000 4.8571 5.0000 khoa_tieng_italia 2024 0.0351 0.0000 4.6675 5.0000 khoa_tieng_nga 1567 0.0530 0.0000 4.4308 5.0000 khoa_tieng_nhat 2448 0.0045 0.0000 4.9261 5.0000 khoa_tieng_phap 2204 0.0181 0.0000 4.3503 5.0000 khoa_tieng_tay_ban_nha 1242 0.0338 0.0000 4.6401 5.0000 khoa_tieng_trung_quoc 5584 0.0136 0.0000 4.7724 6.0000 khoa_viet_nam_hoc 556 0.0701 0.0000 5.0054 6.0000 [SPLIT] train=170,825 validation=115,647 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11176 best_alpha=1.00 robust=+0.01531 meanΔAP=+0.0170 medΔAP=+0.0178 q25=+0.0050 wins=11/14 [SELECT] epoch=02/10 loss=0.10047 best_alpha=0.50 robust=+0.00702 meanΔAP=+0.0074 medΔAP=+0.0077 q25=+0.0009 wins=11/14 [SELECT] epoch=03/10 loss=0.09526 best_alpha=0.50 robust=+0.01065 meanΔAP=+0.0123 medΔAP=+0.0103 q25=+0.0035 wins=13/14 [SELECT] epoch=04/10 loss=0.09418 best_alpha=0.50 robust=+0.00923 meanΔAP=+0.0093 medΔAP=+0.0097 q25=+0.0021 wins=11/14 [SELECT] epoch=05/10 loss=0.08965 best_alpha=0.50 robust=+0.01263 meanΔAP=+0.0150 medΔAP=+0.0140 q25=+0.0019 wins=12/14 [SELECT] epoch=06/10 loss=0.08971 best_alpha=0.50 robust=+0.01276 meanΔAP=+0.0158 medΔAP=+0.0118 q25=+0.0030 wins=11/14 [VARIANT FLAT_RESIDUAL] selected_epochs=1 selected_alpha=1.00 robust=+0.015315 [REFIT] epoch=01/1 loss=0.11756 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.13289 best_alpha=0.50 robust=+0.00829 meanΔAP=+0.0086 medΔAP=+0.0103 q25=+0.0026 wins=11/14 [SELECT] epoch=02/10 loss=0.11974 best_alpha=0.50 robust=+0.00787 meanΔAP=+0.0078 medΔAP=+0.0102 q25=+0.0013 wins=11/14 [SELECT] epoch=03/10 loss=0.11434 best_alpha=0.25 robust=+0.00442 meanΔAP=+0.0047 medΔAP=+0.0050 q25=+0.0028 wins=13/14 [SELECT] epoch=04/10 loss=0.10839 best_alpha=0.50 robust=+0.00482 meanΔAP=+0.0043 medΔAP=+0.0068 q25=+0.0011 wins=12/14 [SELECT] epoch=05/10 loss=0.10817 best_alpha=0.50 robust=+0.00485 meanΔAP=+0.0048 medΔAP=+0.0071 q25=-0.0003 wins=10/14 [SELECT] epoch=06/10 loss=0.10213 best_alpha=0.25 robust=+0.00422 meanΔAP=+0.0040 medΔAP=+0.0051 q25=+0.0010 wins=11/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=0.50 robust=+0.008292 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.14039 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.13417 best_alpha=0.75 robust=+0.00819 meanΔAP=+0.0094 medΔAP=+0.0085 q25=+0.0028 wins=11/14 [SELECT] epoch=02/10 loss=0.12124 best_alpha=0.75 robust=+0.01112 meanΔAP=+0.0127 medΔAP=+0.0120 q25=+0.0067 wins=12/14 [SELECT] epoch=03/10 loss=0.11212 best_alpha=0.50 robust=+0.00563 meanΔAP=+0.0081 medΔAP=+0.0056 q25=+0.0001 wins=10/14 [SELECT] epoch=04/10 loss=0.11076 best_alpha=0.50 robust=+0.00814 meanΔAP=+0.0098 medΔAP=+0.0120 q25=-0.0002 wins=10/14 [SELECT] epoch=05/10 loss=0.10570 best_alpha=0.50 robust=+0.00600 meanΔAP=+0.0082 medΔAP=+0.0078 q25=-0.0033 wins=8/14 [SELECT] epoch=06/10 loss=0.10442 best_alpha=0.50 robust=+0.00629 meanΔAP=+0.0077 medΔAP=+0.0085 q25=-0.0015 wins=8/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=2 selected_alpha=0.75 robust=+0.011116 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/2 loss=0.14019 [REFIT] epoch=02/2 loss=0.12779 SUITE21 TARGET FOLD 2 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 4598 0.0331 0.4729 0.9206 0.0231 0.1413 0.8553 0.4254 khoa_cong_nghe_thong_tin suite21_flat_residual 4598 0.0331 0.4587 0.9033 0.0237 0.1326 0.8026 0.3992 khoa_cong_nghe_thong_tin suite21_temporal_residual 4598 0.0331 0.4720 0.9219 0.0232 0.1413 0.8553 0.4254 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 4598 0.0331 0.4828 0.9226 0.0229 0.1402 0.8487 0.4221 khoa_quan_tri_kinh_doanh_va_du_lich committee 5530 0.0300 0.5498 0.9096 0.0190 0.1257 0.8373 0.3927 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 5530 0.0300 0.5225 0.9078 0.0205 0.1212 0.8072 0.3785 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 5530 0.0300 0.5373 0.9130 0.0202 0.1257 0.8373 0.3927 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 5530 0.0300 0.5309 0.9264 0.0209 0.1293 0.8614 0.4040 khoa_quoc_te_hoc committee 2095 0.0091 0.4660 0.9595 0.0069 0.0430 0.9474 0.1818 khoa_quoc_te_hoc suite21_flat_residual 2095 0.0091 0.4309 0.9289 0.0088 0.0430 0.9474 0.1818 khoa_quoc_te_hoc suite21_temporal_residual 2095 0.0091 0.4203 0.9538 0.0080 0.0430 0.9474 0.1818 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2095 0.0091 0.5078 0.9554 0.0095 0.0430 0.9474 0.1818 khoa_tieng_anh committee 8328 0.0146 0.7490 0.9772 0.0072 0.0702 0.9590 0.2716 khoa_tieng_anh suite21_flat_residual 8328 0.0146 0.7476 0.9762 0.0073 0.0708 0.9672 0.2739 khoa_tieng_anh suite21_temporal_residual 8328 0.0146 0.7477 0.9767 0.0076 0.0708 0.9672 0.2739 khoa_tieng_anh suite21_guarded_temporal_residual 8328 0.0146 0.7568 0.9761 0.0071 0.0702 0.9590 0.2716 khoa_tieng_bo_dao_nha committee 716 0.0419 0.5266 0.9436 0.0263 0.1888 0.9000 0.5133 khoa_tieng_bo_dao_nha suite21_flat_residual 716 0.0419 0.5291 0.9436 0.0270 0.1818 0.8667 0.4943 khoa_tieng_bo_dao_nha suite21_temporal_residual 716 0.0419 0.5098 0.9370 0.0272 0.1748 0.8333 0.4753 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 716 0.0419 0.5512 0.9381 0.0262 0.1748 0.8333 0.4753 khoa_tieng_duc committee 1893 0.0280 0.3930 0.8337 0.0205 0.0897 0.6415 0.2876 khoa_tieng_duc suite21_flat_residual 1893 0.0280 0.3956 0.8431 0.0207 0.1003 0.7170 0.3215 khoa_tieng_duc suite21_temporal_residual 1893 0.0280 0.3970 0.8450 0.0206 0.1029 0.7358 0.3299 khoa_tieng_duc suite21_guarded_temporal_residual 1893 0.0280 0.4180 0.8430 0.0204 0.1029 0.7358 0.3299 khoa_tieng_han_quoc committee 3337 0.0087 0.3792 0.9529 0.0072 0.0390 0.8966 0.1660 khoa_tieng_han_quoc suite21_flat_residual 3337 0.0087 0.3462 0.9552 0.0070 0.0390 0.8966 0.1660 khoa_tieng_han_quoc suite21_temporal_residual 3337 0.0087 0.3627 0.9526 0.0071 0.0405 0.9310 0.1724 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3337 0.0087 0.4043 0.9545 0.0070 0.0405 0.9310 0.1724 khoa_tieng_italia committee 2024 0.0351 0.6839 0.9550 0.0182 0.1605 0.9155 0.4717 khoa_tieng_italia suite21_flat_residual 2024 0.0351 0.6949 0.9499 0.0203 0.1580 0.9014 0.4644 khoa_tieng_italia suite21_temporal_residual 2024 0.0351 0.6641 0.9516 0.0192 0.1580 0.9014 0.4644 khoa_tieng_italia suite21_guarded_temporal_residual 2024 0.0351 0.6686 0.9456 0.0195 0.1605 0.9155 0.4717 khoa_tieng_nga committee 1567 0.0530 0.7595 0.9786 0.0228 0.2620 0.9880 0.6357 khoa_tieng_nga suite21_flat_residual 1567 0.0530 0.7481 0.9749 0.0249 0.2588 0.9759 0.6279 khoa_tieng_nga suite21_temporal_residual 1567 0.0530 0.7816 0.9812 0.0221 0.2620 0.9880 0.6357 khoa_tieng_nga suite21_guarded_temporal_residual 1567 0.0530 0.7956 0.9800 0.0224 0.2620 0.9880 0.6357 khoa_tieng_nhat committee 2448 0.0045 0.5422 0.9638 0.0029 0.0224 1.0000 0.1030 khoa_tieng_nhat suite21_flat_residual 2448 0.0045 0.5276 0.9757 0.0031 0.0224 1.0000 0.1030 khoa_tieng_nhat suite21_temporal_residual 2448 0.0045 0.5467 0.9745 0.0029 0.0224 1.0000 0.1030 khoa_tieng_nhat suite21_guarded_temporal_residual 2448 0.0045 0.5820 0.9757 0.0030 0.0224 1.0000 0.1030 khoa_tieng_phap committee 2204 0.0181 0.5846 0.9370 0.0101 0.0816 0.9000 0.2995 khoa_tieng_phap suite21_flat_residual 2204 0.0181 0.5460 0.9404 0.0104 0.0816 0.9000 0.2995 khoa_tieng_phap suite21_temporal_residual 2204 0.0181 0.5940 0.9448 0.0099 0.0816 0.9000 0.2995 khoa_tieng_phap suite21_guarded_temporal_residual 2204 0.0181 0.5551 0.9410 0.0104 0.0816 0.9000 0.2995 khoa_tieng_tay_ban_nha committee 1242 0.0338 0.3776 0.9312 0.0252 0.1411 0.8333 0.4207 khoa_tieng_tay_ban_nha suite21_flat_residual 1242 0.0338 0.4035 0.9355 0.0250 0.1492 0.8810 0.4447 khoa_tieng_tay_ban_nha suite21_temporal_residual 1242 0.0338 0.3867 0.9315 0.0253 0.1452 0.8571 0.4327 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1242 0.0338 0.3902 0.9289 0.0251 0.1452 0.8571 0.4327 khoa_tieng_trung_quoc committee 5584 0.0136 0.0910 0.8927 0.0133 0.0555 0.8158 0.2182 khoa_tieng_trung_quoc suite21_flat_residual 5584 0.0136 0.1161 0.9041 0.0128 0.0582 0.8553 0.2287 khoa_tieng_trung_quoc suite21_temporal_residual 5584 0.0136 0.1132 0.9017 0.0129 0.0573 0.8421 0.2252 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 5584 0.0136 0.1090 0.9018 0.0129 0.0573 0.8421 0.2252 khoa_viet_nam_hoc committee 556 0.0701 0.4830 0.8092 0.0464 0.2342 0.6667 0.4869 khoa_viet_nam_hoc suite21_flat_residual 556 0.0701 0.4611 0.8214 0.0479 0.2342 0.6667 0.4869 khoa_viet_nam_hoc suite21_temporal_residual 556 0.0701 0.4826 0.8206 0.0469 0.2342 0.6667 0.4869 khoa_viet_nam_hoc suite21_guarded_temporal_residual 556 0.0701 0.4338 0.8081 0.0496 0.2342 0.6667 0.4869 SUITE21 TARGET FOLD 2 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.4729 0.4828 0.0100 0.0112 0.9804 0.8576 0.0448 0.0096 khoa_quan_tri_kinh_doanh_va_du_lich 0.5498 0.5309 -0.0190 0.0152 0.9826 0.8237 0.0659 0.0128 khoa_quoc_te_hoc 0.4660 0.5078 0.0418 0.0174 0.9682 0.8115 0.1196 0.0162 khoa_tieng_anh 0.7490 0.7568 0.0078 0.0027 0.9966 0.9322 0.0108 -0.0007 khoa_tieng_bo_dao_nha 0.5266 0.5512 0.0246 0.0117 0.9824 0.8182 0.0538 0.0083 khoa_tieng_duc 0.3930 0.4180 0.0249 0.0036 0.9823 0.8918 0.0101 0.0025 khoa_tieng_han_quoc 0.3792 0.4043 0.0251 0.0012 0.9810 0.9820 -0.0154 -0.0007 khoa_tieng_italia 0.6839 0.6686 -0.0153 0.0114 0.9622 0.8617 0.0343 0.0082 khoa_tieng_nga 0.7595 0.7956 0.0361 0.0217 0.9675 0.8339 0.0904 0.0161 khoa_tieng_nhat 0.5422 0.5820 0.0398 0.0014 0.9931 0.9653 0.0314 -0.0001 khoa_tieng_phap 0.5846 0.5551 -0.0295 0.0039 0.9850 0.9252 0.0425 0.0016 khoa_tieng_tay_ban_nha 0.3776 0.3902 0.0126 0.0063 0.9703 0.8427 0.0224 0.0032 khoa_tieng_trung_quoc 0.0910 0.1090 0.0180 0.0035 0.9982 0.9749 -0.0003 -0.0022 khoa_viet_nam_hoc 0.4830 0.4338 -0.0491 0.0065 0.9941 0.8468 -0.0104 0.0046 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 3 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=329,579 failures=6,495 (1.97%) target=47,420 failures=1,103 (2.33%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 6277 0.0347 0.0000 5.0899 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 6491 0.0322 0.0000 5.0390 6.0000 khoa_quoc_te_hoc 2665 0.0098 0.0000 4.9786 6.0000 khoa_tieng_anh 7279 0.0139 0.0000 4.3367 4.0000 khoa_tieng_bo_dao_nha 1032 0.0484 0.0000 4.4535 4.0000 khoa_tieng_duc 1909 0.0246 0.0000 4.5359 4.0000 khoa_tieng_han_quoc 3235 0.0130 0.0000 4.4652 4.0000 khoa_tieng_italia 2145 0.0317 0.0000 4.4490 4.0000 khoa_tieng_nga 1996 0.0461 0.0000 4.0902 4.0000 khoa_tieng_nhat 2588 0.0066 0.0000 4.2400 4.0000 khoa_tieng_phap 2693 0.0160 0.0000 4.2001 4.0000 khoa_tieng_tay_ban_nha 1417 0.0381 0.0000 4.2315 4.0000 khoa_tieng_trung_quoc 6920 0.0149 0.0000 4.3497 4.0000 khoa_viet_nam_hoc 773 0.0427 0.0000 5.0918 6.0000 [SPLIT] train=219,176 validation=110,403 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11197 best_alpha=0.50 robust=+0.00409 meanΔAP=+0.0049 medΔAP=+0.0034 q25=+0.0009 wins=11/14 [SELECT] epoch=02/10 loss=0.10361 best_alpha=0.25 robust=+0.00377 meanΔAP=+0.0040 medΔAP=+0.0050 q25=+0.0003 wins=10/14 [SELECT] epoch=03/10 loss=0.09872 best_alpha=0.25 robust=+0.00574 meanΔAP=+0.0065 medΔAP=+0.0057 q25=+0.0044 wins=11/14 [SELECT] epoch=04/10 loss=0.09320 best_alpha=0.50 robust=+0.00736 meanΔAP=+0.0066 medΔAP=+0.0130 q25=-0.0021 wins=10/14 [SELECT] epoch=05/10 loss=0.08963 best_alpha=0.25 robust=+0.00448 meanΔAP=+0.0049 medΔAP=+0.0071 q25=-0.0011 wins=9/14 [SELECT] epoch=06/10 loss=0.08845 best_alpha=0.25 robust=+0.00504 meanΔAP=+0.0056 medΔAP=+0.0065 q25=+0.0014 wins=11/14 [SELECT] epoch=07/10 loss=0.08752 best_alpha=0.25 robust=+0.00507 meanΔAP=+0.0053 medΔAP=+0.0075 q25=+0.0020 wins=11/14 [VARIANT FLAT_RESIDUAL] selected_epochs=4 selected_alpha=0.50 robust=+0.007360 [REFIT] epoch=01/4 loss=0.11688 [REFIT] epoch=04/4 loss=0.10237 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.13501 best_alpha=0.50 robust=+0.00320 meanΔAP=+0.0021 medΔAP=+0.0061 q25=-0.0023 wins=10/14 [SELECT] epoch=02/10 loss=0.12301 best_alpha=0.25 robust=+0.00635 meanΔAP=+0.0072 medΔAP=+0.0066 q25=+0.0033 wins=12/14 [SELECT] epoch=03/10 loss=0.11461 best_alpha=0.25 robust=+0.00369 meanΔAP=+0.0047 medΔAP=+0.0046 q25=-0.0008 wins=9/14 [SELECT] epoch=04/10 loss=0.10945 best_alpha=0.25 robust=+0.00365 meanΔAP=+0.0037 medΔAP=+0.0045 q25=+0.0014 wins=11/14 [SELECT] epoch=05/10 loss=0.10780 best_alpha=0.50 robust=+0.00696 meanΔAP=+0.0086 medΔAP=+0.0072 q25=+0.0013 wins=11/14 [SELECT] epoch=06/10 loss=0.10598 best_alpha=0.50 robust=+0.00802 meanΔAP=+0.0101 medΔAP=+0.0085 q25=+0.0024 wins=12/14 [SELECT] epoch=07/10 loss=0.10503 best_alpha=0.50 robust=+0.00967 meanΔAP=+0.0087 medΔAP=+0.0120 q25=+0.0097 wins=11/14 [SELECT] epoch=08/10 loss=0.10117 best_alpha=0.50 robust=+0.01070 meanΔAP=+0.0124 medΔAP=+0.0119 q25=+0.0051 wins=11/14 [SELECT] epoch=09/10 loss=0.09922 best_alpha=0.25 robust=+0.00509 meanΔAP=+0.0061 medΔAP=+0.0062 q25=+0.0021 wins=11/14 [SELECT] epoch=10/10 loss=0.09762 best_alpha=0.25 robust=+0.00598 meanΔAP=+0.0064 medΔAP=+0.0069 q25=+0.0028 wins=11/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=8 selected_alpha=0.50 robust=+0.010699 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/8 loss=0.13907 [REFIT] epoch=08/8 loss=0.11148 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.13538 best_alpha=0.50 robust=+0.00608 meanΔAP=+0.0066 medΔAP=+0.0073 q25=-0.0003 wins=10/14 [SELECT] epoch=02/10 loss=0.12135 best_alpha=0.50 robust=+0.00577 meanΔAP=+0.0063 medΔAP=+0.0095 q25=-0.0060 wins=10/14 [SELECT] epoch=03/10 loss=0.11330 best_alpha=0.50 robust=+0.00491 meanΔAP=+0.0045 medΔAP=+0.0059 q25=+0.0033 wins=11/14 [SELECT] epoch=04/10 loss=0.11095 best_alpha=0.50 robust=+0.00519 meanΔAP=+0.0055 medΔAP=+0.0073 q25=+0.0018 wins=11/14 [SELECT] epoch=05/10 loss=0.10984 best_alpha=0.25 robust=+0.00301 meanΔAP=+0.0031 medΔAP=+0.0037 q25=+0.0020 wins=11/14 [SELECT] epoch=06/10 loss=0.10656 best_alpha=0.25 robust=+0.00125 meanΔAP=+0.0002 medΔAP=+0.0037 q25=+0.0022 wins=12/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=0.50 robust=+0.006083 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.14008 SUITE21 TARGET FOLD 3 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 6277 0.0347 0.2738 0.8506 0.0305 0.1283 0.7385 0.3785 khoa_cong_nghe_thong_tin suite21_flat_residual 6277 0.0347 0.2792 0.8543 0.0305 0.1299 0.7477 0.3832 khoa_cong_nghe_thong_tin suite21_temporal_residual 6277 0.0347 0.2928 0.8641 0.0304 0.1331 0.7661 0.3926 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 6277 0.0347 0.2828 0.8527 0.0305 0.1259 0.7248 0.3714 khoa_quan_tri_kinh_doanh_va_du_lich committee 6491 0.0322 0.3846 0.9126 0.0259 0.1364 0.8469 0.4147 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 6491 0.0322 0.3830 0.9119 0.0260 0.1348 0.8373 0.4100 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 6491 0.0322 0.4005 0.9177 0.0251 0.1379 0.8565 0.4194 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 6491 0.0322 0.3966 0.9133 0.0256 0.1371 0.8517 0.4171 khoa_quoc_te_hoc committee 2665 0.0098 0.1991 0.8143 0.0091 0.0300 0.6154 0.1256 khoa_quoc_te_hoc suite21_flat_residual 2665 0.0098 0.1487 0.8186 0.0092 0.0300 0.6154 0.1256 khoa_quoc_te_hoc suite21_temporal_residual 2665 0.0098 0.1645 0.8211 0.0095 0.0300 0.6154 0.1256 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2665 0.0098 0.1965 0.8166 0.0092 0.0300 0.6154 0.1256 khoa_tieng_anh committee 7279 0.0139 0.3468 0.9408 0.0112 0.0646 0.9307 0.2527 khoa_tieng_anh suite21_flat_residual 7279 0.0139 0.3518 0.9425 0.0111 0.0639 0.9208 0.2500 khoa_tieng_anh suite21_temporal_residual 7279 0.0139 0.3842 0.9448 0.0108 0.0646 0.9307 0.2527 khoa_tieng_anh suite21_guarded_temporal_residual 7279 0.0139 0.3836 0.9477 0.0109 0.0652 0.9406 0.2554 khoa_tieng_bo_dao_nha committee 1032 0.0484 0.3489 0.9100 0.0384 0.2087 0.8600 0.5296 khoa_tieng_bo_dao_nha suite21_flat_residual 1032 0.0484 0.3508 0.9096 0.0381 0.2087 0.8600 0.5296 khoa_tieng_bo_dao_nha suite21_temporal_residual 1032 0.0484 0.4636 0.9151 0.0354 0.1893 0.7800 0.4803 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 1032 0.0484 0.3969 0.9197 0.0363 0.2087 0.8600 0.5296 khoa_tieng_duc committee 1909 0.0246 0.2476 0.8529 0.0223 0.0942 0.7660 0.3158 khoa_tieng_duc suite21_flat_residual 1909 0.0246 0.2471 0.8616 0.0221 0.0942 0.7660 0.3158 khoa_tieng_duc suite21_temporal_residual 1909 0.0246 0.2460 0.8774 0.0217 0.0942 0.7660 0.3158 khoa_tieng_duc suite21_guarded_temporal_residual 1909 0.0246 0.2411 0.8658 0.0229 0.0969 0.7872 0.3246 khoa_tieng_han_quoc committee 3235 0.0130 0.2147 0.8411 0.0119 0.0433 0.6667 0.1718 khoa_tieng_han_quoc suite21_flat_residual 3235 0.0130 0.2156 0.8439 0.0119 0.0448 0.6905 0.1779 khoa_tieng_han_quoc suite21_temporal_residual 3235 0.0130 0.2038 0.8411 0.0115 0.0448 0.6905 0.1779 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3235 0.0130 0.2299 0.8490 0.0115 0.0464 0.7143 0.1840 khoa_tieng_italia committee 2145 0.0317 0.2073 0.8333 0.0294 0.1002 0.6324 0.3067 khoa_tieng_italia suite21_flat_residual 2145 0.0317 0.2058 0.8366 0.0293 0.1002 0.6324 0.3067 khoa_tieng_italia suite21_temporal_residual 2145 0.0317 0.1845 0.8336 0.0304 0.0886 0.5588 0.2710 khoa_tieng_italia suite21_guarded_temporal_residual 2145 0.0317 0.2070 0.8350 0.0300 0.0932 0.5882 0.2853 khoa_tieng_nga committee 1996 0.0461 0.4220 0.9314 0.0345 0.1980 0.8587 0.5150 khoa_tieng_nga suite21_flat_residual 1996 0.0461 0.4290 0.9348 0.0340 0.1980 0.8587 0.5150 khoa_tieng_nga suite21_temporal_residual 1996 0.0461 0.4283 0.9336 0.0334 0.1980 0.8587 0.5150 khoa_tieng_nga suite21_guarded_temporal_residual 1996 0.0461 0.4568 0.9423 0.0331 0.2080 0.9022 0.5411 khoa_tieng_nhat committee 2588 0.0066 0.3773 0.9403 0.0058 0.0309 0.9412 0.1365 khoa_tieng_nhat suite21_flat_residual 2588 0.0066 0.3613 0.9411 0.0057 0.0309 0.9412 0.1365 khoa_tieng_nhat suite21_temporal_residual 2588 0.0066 0.3486 0.9345 0.0056 0.0309 0.9412 0.1365 khoa_tieng_nhat suite21_guarded_temporal_residual 2588 0.0066 0.3626 0.9404 0.0056 0.0309 0.9412 0.1365 khoa_tieng_phap committee 2693 0.0160 0.1018 0.6932 0.0166 0.0297 0.3721 0.1125 khoa_tieng_phap suite21_flat_residual 2693 0.0160 0.1233 0.6987 0.0163 0.0334 0.4186 0.1266 khoa_tieng_phap suite21_temporal_residual 2693 0.0160 0.0962 0.7161 0.0189 0.0353 0.4419 0.1336 khoa_tieng_phap suite21_guarded_temporal_residual 2693 0.0160 0.1084 0.7164 0.0170 0.0353 0.4419 0.1336 khoa_tieng_tay_ban_nha committee 1417 0.0381 0.4401 0.9124 0.0310 0.1555 0.8148 0.4409 khoa_tieng_tay_ban_nha suite21_flat_residual 1417 0.0381 0.4584 0.9196 0.0303 0.1590 0.8333 0.4509 khoa_tieng_tay_ban_nha suite21_temporal_residual 1417 0.0381 0.5075 0.9154 0.0284 0.1519 0.7963 0.4309 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1417 0.0381 0.4682 0.9235 0.0298 0.1731 0.9074 0.4910 khoa_tieng_trung_quoc committee 6920 0.0149 0.1204 0.8968 0.0141 0.0665 0.8932 0.2561 khoa_tieng_trung_quoc suite21_flat_residual 6920 0.0149 0.1231 0.8962 0.0140 0.0672 0.9029 0.2589 khoa_tieng_trung_quoc suite21_temporal_residual 6920 0.0149 0.1311 0.8927 0.0138 0.0672 0.9029 0.2589 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 6920 0.0149 0.1365 0.8998 0.0139 0.0665 0.8932 0.2561 khoa_viet_nam_hoc committee 773 0.0427 0.4887 0.9212 0.0325 0.1806 0.8485 0.4878 khoa_viet_nam_hoc suite21_flat_residual 773 0.0427 0.4777 0.9185 0.0334 0.1806 0.8485 0.4878 khoa_viet_nam_hoc suite21_temporal_residual 773 0.0427 0.4130 0.9331 0.0329 0.1806 0.8485 0.4878 khoa_viet_nam_hoc suite21_guarded_temporal_residual 773 0.0427 0.4905 0.9183 0.0343 0.1806 0.8485 0.4878 SUITE21 TARGET FOLD 3 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.2738 0.2828 0.0089 0.0046 0.9977 0.9554 0.0127 -0.0009 khoa_quan_tri_kinh_doanh_va_du_lich 0.3846 0.3966 0.0120 0.0049 0.9978 0.9499 0.0178 -0.0008 khoa_quoc_te_hoc 0.1991 0.1965 -0.0026 0.0023 0.9977 0.9456 0.0236 0.0004 khoa_tieng_anh 0.3468 0.3836 0.0368 0.0027 0.9983 0.9519 0.0283 -0.0003 khoa_tieng_bo_dao_nha 0.3489 0.3969 0.0480 0.0072 0.9977 0.9466 0.0371 -0.0010 khoa_tieng_duc 0.2476 0.2411 -0.0064 0.0068 0.9928 0.9110 0.0210 0.0005 khoa_tieng_han_quoc 0.2147 0.2299 0.0152 0.0032 0.9981 0.9444 0.0280 -0.0009 khoa_tieng_italia 0.2073 0.2070 -0.0004 0.0089 0.9912 0.8881 0.0161 0.0027 khoa_tieng_nga 0.4220 0.4568 0.0348 0.0067 0.9882 0.9424 0.0299 -0.0005 khoa_tieng_nhat 0.3773 0.3626 -0.0147 0.0020 0.9990 0.9633 0.0238 -0.0009 khoa_tieng_phap 0.1018 0.1084 0.0065 0.0064 0.9933 0.8738 0.0230 0.0015 khoa_tieng_tay_ban_nha 0.4401 0.4682 0.0281 0.0060 0.9947 0.9152 0.0219 -0.0022 khoa_tieng_trung_quoc 0.1204 0.1365 0.0161 0.0037 0.9981 0.9581 0.0082 -0.0021 khoa_viet_nam_hoc 0.4887 0.4905 0.0018 0.0118 0.9937 0.9419 0.0318 0.0102 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 4 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=376,999 failures=7,598 (2.02%) target=41,376 failures=1,100 (2.66%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 5072 0.0481 0.0000 5.2474 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 6440 0.0374 0.0000 5.2509 6.0000 khoa_quoc_te_hoc 2202 0.0159 0.0000 5.2661 6.0000 khoa_tieng_anh 6636 0.0102 0.0000 4.7560 5.0000 khoa_tieng_bo_dao_nha 869 0.0736 0.0000 5.0483 5.0000 khoa_tieng_duc 1746 0.0252 0.0000 5.0481 5.0000 khoa_tieng_han_quoc 3351 0.0113 0.0000 4.9728 5.0000 khoa_tieng_italia 1795 0.0189 0.0000 4.8607 5.0000 khoa_tieng_nga 1659 0.0687 0.0000 4.5955 5.0000 khoa_tieng_nhat 2413 0.0157 0.0000 4.9581 5.0000 khoa_tieng_phap 2275 0.0084 0.0000 4.5433 5.0000 khoa_tieng_tay_ban_nha 1306 0.0299 0.0000 4.6332 5.0000 khoa_tieng_trung_quoc 5107 0.0204 0.0000 4.6017 5.0000 khoa_viet_nam_hoc 505 0.0356 0.0000 5.2634 6.0000 [SPLIT] train=254,508 validation=122,491 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11184 best_alpha=0.25 robust=+0.00154 meanΔAP=+0.0020 medΔAP=+0.0018 q25=-0.0017 wins=9/14 [SELECT] epoch=02/10 loss=0.10375 best_alpha=0.25 robust=+0.00736 meanΔAP=+0.0081 medΔAP=+0.0091 q25=+0.0032 wins=12/14 [SELECT] epoch=03/10 loss=0.09646 best_alpha=0.25 robust=+0.00897 meanΔAP=+0.0098 medΔAP=+0.0116 q25=+0.0036 wins=11/14 [SELECT] epoch=04/10 loss=0.09542 best_alpha=0.25 robust=+0.00679 meanΔAP=+0.0082 medΔAP=+0.0078 q25=+0.0022 wins=12/14 [SELECT] epoch=05/10 loss=0.09076 best_alpha=0.25 robust=+0.00778 meanΔAP=+0.0084 medΔAP=+0.0098 q25=+0.0044 wins=13/14 [SELECT] epoch=06/10 loss=0.08929 best_alpha=0.25 robust=+0.00381 meanΔAP=+0.0048 medΔAP=+0.0044 q25=+0.0001 wins=10/14 [VARIANT FLAT_RESIDUAL] selected_epochs=3 selected_alpha=0.25 robust=+0.008972 [REFIT] epoch=01/3 loss=0.11789 [REFIT] epoch=03/3 loss=0.10530 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.13702 best_alpha=0.75 robust=+0.01054 meanΔAP=+0.0108 medΔAP=+0.0137 q25=+0.0019 wins=11/14 [SELECT] epoch=02/10 loss=0.12207 best_alpha=0.50 robust=+0.00976 meanΔAP=+0.0088 medΔAP=+0.0136 q25=+0.0054 wins=11/14 [SELECT] epoch=03/10 loss=0.11871 best_alpha=0.50 robust=+0.01016 meanΔAP=+0.0127 medΔAP=+0.0088 q25=+0.0068 wins=11/14 [SELECT] epoch=04/10 loss=0.11044 best_alpha=0.50 robust=+0.00538 meanΔAP=+0.0074 medΔAP=+0.0065 q25=-0.0027 wins=9/14 [SELECT] epoch=05/10 loss=0.10859 best_alpha=0.25 robust=+0.00514 meanΔAP=+0.0061 medΔAP=+0.0061 q25=+0.0008 wins=10/14 [SELECT] epoch=06/10 loss=0.10519 best_alpha=0.50 robust=+0.00855 meanΔAP=+0.0107 medΔAP=+0.0092 q25=+0.0007 wins=10/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=0.75 robust=+0.010542 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.14109 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.13420 best_alpha=0.50 robust=+0.00768 meanΔAP=+0.0095 medΔAP=+0.0098 q25=-0.0021 wins=10/14 [SELECT] epoch=02/10 loss=0.12478 best_alpha=0.50 robust=+0.00581 meanΔAP=+0.0070 medΔAP=+0.0080 q25=-0.0021 wins=10/14 [SELECT] epoch=03/10 loss=0.11490 best_alpha=0.50 robust=+0.01220 meanΔAP=+0.0132 medΔAP=+0.0170 q25=+0.0050 wins=11/14 [SELECT] epoch=04/10 loss=0.11142 best_alpha=0.50 robust=+0.00733 meanΔAP=+0.0077 medΔAP=+0.0099 q25=+0.0037 wins=12/14 [SELECT] epoch=05/10 loss=0.10858 best_alpha=0.50 robust=+0.00980 meanΔAP=+0.0098 medΔAP=+0.0138 q25=+0.0026 wins=11/14 [SELECT] epoch=06/10 loss=0.10735 best_alpha=0.50 robust=+0.00707 meanΔAP=+0.0072 medΔAP=+0.0094 q25=+0.0036 wins=12/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=3 selected_alpha=0.50 robust=+0.012200 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/3 loss=0.13916 [REFIT] epoch=03/3 loss=0.12436 SUITE21 TARGET FOLD 4 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 5072 0.0481 0.3788 0.8880 0.0374 0.1893 0.7869 0.4824 khoa_cong_nghe_thong_tin suite21_flat_residual 5072 0.0481 0.3769 0.8876 0.0375 0.1884 0.7828 0.4799 khoa_cong_nghe_thong_tin suite21_temporal_residual 5072 0.0481 0.3835 0.8937 0.0375 0.1884 0.7828 0.4799 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 5072 0.0481 0.3885 0.8932 0.0370 0.1903 0.7910 0.4849 khoa_quan_tri_kinh_doanh_va_du_lich committee 6440 0.0374 0.4325 0.9205 0.0277 0.1630 0.8714 0.4663 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 6440 0.0374 0.4300 0.9203 0.0278 0.1623 0.8672 0.4640 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 6440 0.0374 0.4294 0.9243 0.0285 0.1638 0.8755 0.4685 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 6440 0.0374 0.4388 0.9253 0.0275 0.1646 0.8797 0.4707 khoa_quoc_te_hoc committee 2202 0.0159 0.0971 0.8807 0.0163 0.0659 0.8286 0.2500 khoa_quoc_te_hoc suite21_flat_residual 2202 0.0159 0.0974 0.8770 0.0164 0.0614 0.7714 0.2328 khoa_quoc_te_hoc suite21_temporal_residual 2202 0.0159 0.0980 0.8806 0.0171 0.0636 0.8000 0.2414 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2202 0.0159 0.0976 0.8909 0.0172 0.0682 0.8571 0.2586 khoa_tieng_anh committee 6636 0.0102 0.4497 0.9221 0.0075 0.0422 0.8235 0.1751 khoa_tieng_anh suite21_flat_residual 6636 0.0102 0.4503 0.9223 0.0075 0.0422 0.8235 0.1751 khoa_tieng_anh suite21_temporal_residual 6636 0.0102 0.4691 0.9274 0.0074 0.0414 0.8088 0.1720 khoa_tieng_anh suite21_guarded_temporal_residual 6636 0.0102 0.4608 0.9260 0.0074 0.0422 0.8235 0.1751 khoa_tieng_bo_dao_nha committee 869 0.0736 0.6562 0.9585 0.0374 0.3448 0.9375 0.6977 khoa_tieng_bo_dao_nha suite21_flat_residual 869 0.0736 0.6538 0.9581 0.0373 0.3448 0.9375 0.6977 khoa_tieng_bo_dao_nha suite21_temporal_residual 869 0.0736 0.6774 0.9592 0.0361 0.3448 0.9375 0.6977 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 869 0.0736 0.6607 0.9606 0.0372 0.3563 0.9688 0.7209 khoa_tieng_duc committee 1746 0.0252 0.4463 0.9197 0.0201 0.1117 0.8864 0.3714 khoa_tieng_duc suite21_flat_residual 1746 0.0252 0.4433 0.9176 0.0203 0.1117 0.8864 0.3714 khoa_tieng_duc suite21_temporal_residual 1746 0.0252 0.4400 0.9236 0.0211 0.1117 0.8864 0.3714 khoa_tieng_duc suite21_guarded_temporal_residual 1746 0.0252 0.4659 0.9225 0.0203 0.1117 0.8864 0.3714 khoa_tieng_han_quoc committee 3351 0.0113 0.4396 0.9574 0.0084 0.0522 0.9211 0.2129 khoa_tieng_han_quoc suite21_flat_residual 3351 0.0113 0.4692 0.9583 0.0082 0.0522 0.9211 0.2129 khoa_tieng_han_quoc suite21_temporal_residual 3351 0.0113 0.4814 0.9606 0.0082 0.0537 0.9474 0.2190 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3351 0.0113 0.4741 0.9589 0.0084 0.0522 0.9211 0.2129 khoa_tieng_italia committee 1795 0.0189 0.3979 0.9106 0.0157 0.0836 0.8824 0.3030 khoa_tieng_italia suite21_flat_residual 1795 0.0189 0.4111 0.9140 0.0155 0.0808 0.8529 0.2929 khoa_tieng_italia suite21_temporal_residual 1795 0.0189 0.4301 0.9282 0.0157 0.0808 0.8529 0.2929 khoa_tieng_italia suite21_guarded_temporal_residual 1795 0.0189 0.4047 0.9194 0.0156 0.0808 0.8529 0.2929 khoa_tieng_nga committee 1659 0.0687 0.6177 0.9271 0.0385 0.2861 0.8333 0.6028 khoa_tieng_nga suite21_flat_residual 1659 0.0687 0.6190 0.9277 0.0385 0.2861 0.8333 0.6028 khoa_tieng_nga suite21_temporal_residual 1659 0.0687 0.6109 0.9291 0.0392 0.2922 0.8509 0.6155 khoa_tieng_nga suite21_guarded_temporal_residual 1659 0.0687 0.6093 0.9288 0.0390 0.2952 0.8596 0.6218 khoa_tieng_nhat committee 2413 0.0157 0.1667 0.8613 0.0157 0.0580 0.7368 0.2205 khoa_tieng_nhat suite21_flat_residual 2413 0.0157 0.1765 0.8649 0.0157 0.0580 0.7368 0.2205 khoa_tieng_nhat suite21_temporal_residual 2413 0.0157 0.1777 0.8594 0.0155 0.0559 0.7105 0.2126 khoa_tieng_nhat suite21_guarded_temporal_residual 2413 0.0157 0.1770 0.8692 0.0160 0.0642 0.8158 0.2441 khoa_tieng_phap committee 2275 0.0084 0.2944 0.8394 0.0092 0.0308 0.7368 0.1318 khoa_tieng_phap suite21_flat_residual 2275 0.0084 0.2997 0.8417 0.0091 0.0308 0.7368 0.1318 khoa_tieng_phap suite21_temporal_residual 2275 0.0084 0.3007 0.8476 0.0095 0.0330 0.7895 0.1412 khoa_tieng_phap suite21_guarded_temporal_residual 2275 0.0084 0.3239 0.8447 0.0096 0.0308 0.7368 0.1318 khoa_tieng_tay_ban_nha committee 1306 0.0299 0.5985 0.9100 0.0233 0.1264 0.8462 0.3957 khoa_tieng_tay_ban_nha suite21_flat_residual 1306 0.0299 0.5991 0.9107 0.0236 0.1264 0.8462 0.3957 khoa_tieng_tay_ban_nha suite21_temporal_residual 1306 0.0299 0.5980 0.9148 0.0240 0.1264 0.8462 0.3957 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1306 0.0299 0.5993 0.9145 0.0237 0.1303 0.8718 0.4077 khoa_tieng_trung_quoc committee 5107 0.0204 0.4713 0.9579 0.0152 0.0960 0.9423 0.3410 khoa_tieng_trung_quoc suite21_flat_residual 5107 0.0204 0.4749 0.9587 0.0152 0.0960 0.9423 0.3410 khoa_tieng_trung_quoc suite21_temporal_residual 5107 0.0204 0.4843 0.9611 0.0145 0.0950 0.9327 0.3375 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 5107 0.0204 0.4659 0.9598 0.0149 0.0960 0.9423 0.3410 khoa_viet_nam_hoc committee 505 0.0356 0.5192 0.9606 0.0251 0.1683 0.9444 0.4913 khoa_viet_nam_hoc suite21_flat_residual 505 0.0356 0.5162 0.9604 0.0256 0.1683 0.9444 0.4913 khoa_viet_nam_hoc suite21_temporal_residual 505 0.0356 0.5274 0.9613 0.0257 0.1683 0.9444 0.4913 khoa_viet_nam_hoc suite21_guarded_temporal_residual 505 0.0356 0.5480 0.9580 0.0260 0.1683 0.9444 0.4913 SUITE21 TARGET FOLD 4 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.3788 0.3885 0.0098 0.0058 0.9947 0.9300 0.0104 -0.0014 khoa_quan_tri_kinh_doanh_va_du_lich 0.4325 0.4388 0.0063 0.0051 0.9970 0.9356 0.0172 -0.0007 khoa_quoc_te_hoc 0.0971 0.0976 0.0005 0.0044 0.9954 0.9227 0.0147 0.0018 khoa_tieng_anh 0.4497 0.4608 0.0111 0.0024 0.9992 0.9736 0.0115 -0.0013 khoa_tieng_bo_dao_nha 0.6562 0.6607 0.0044 0.0053 0.9966 0.9425 0.0113 0.0021 khoa_tieng_duc 0.4463 0.4659 0.0196 0.0045 0.9913 0.8596 0.0199 0.0017 khoa_tieng_han_quoc 0.4396 0.4741 0.0345 0.0017 0.9986 0.9612 0.0195 0.0004 khoa_tieng_italia 0.3979 0.4047 0.0067 0.0037 0.9937 0.9499 0.0195 0.0011 khoa_tieng_nga 0.6177 0.6093 -0.0084 0.0057 0.9950 0.9578 0.0190 0.0033 khoa_tieng_nhat 0.1667 0.1770 0.0103 0.0014 0.9990 0.9689 0.0068 0.0006 khoa_tieng_phap 0.2944 0.3239 0.0295 0.0031 0.9954 0.9253 0.0389 0.0012 khoa_tieng_tay_ban_nha 0.5985 0.5993 0.0009 0.0032 0.9928 0.9042 0.0071 0.0002 khoa_tieng_trung_quoc 0.4713 0.4659 -0.0054 0.0035 0.9994 0.9608 0.0128 -0.0016 khoa_viet_nam_hoc 0.5192 0.5480 0.0288 0.0052 0.9975 0.9406 0.0116 0.0001 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 5 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=419,467 failures=8,736 (2.08%) target=50,698 failures=1,797 (3.54%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 6398 0.0549 0.0000 5.1143 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 7604 0.0535 0.0000 5.0705 6.0000 khoa_quoc_te_hoc 2674 0.0217 0.0000 4.8747 6.0000 khoa_tieng_anh 8345 0.0158 0.0000 4.3324 4.0000 khoa_tieng_bo_dao_nha 1138 0.0703 0.0000 4.5132 4.0000 khoa_tieng_duc 1975 0.0299 0.0000 4.2218 4.0000 khoa_tieng_han_quoc 3573 0.0188 0.0000 4.4176 4.0000 khoa_tieng_italia 2129 0.0390 0.0000 4.4462 4.0000 khoa_tieng_nga 2047 0.0655 0.0000 4.4787 6.0000 khoa_tieng_nhat 2593 0.0266 0.0000 4.0956 4.0000 khoa_tieng_phap 2983 0.0191 0.0000 4.0865 4.0000 khoa_tieng_tay_ban_nha 1606 0.0523 0.0000 4.1619 4.0000 khoa_tieng_trung_quoc 6995 0.0222 0.0000 4.4536 4.0000 khoa_viet_nam_hoc 638 0.0956 0.0000 4.3041 4.0000 [SPLIT] train=313,390 validation=106,077 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11706 best_alpha=0.75 robust=+0.00842 meanΔAP=+0.0106 medΔAP=+0.0099 q25=+0.0030 wins=11/14 [SELECT] epoch=02/10 loss=0.10870 best_alpha=0.50 robust=+0.00637 meanΔAP=+0.0095 medΔAP=+0.0077 q25=-0.0028 wins=9/14 [SELECT] epoch=03/10 loss=0.10227 best_alpha=0.75 robust=+0.00653 meanΔAP=+0.0076 medΔAP=+0.0083 q25=+0.0021 wins=11/14 [SELECT] epoch=04/10 loss=0.10078 best_alpha=0.50 robust=+0.00444 meanΔAP=+0.0057 medΔAP=+0.0049 q25=+0.0015 wins=11/14 [SELECT] epoch=05/10 loss=0.09663 best_alpha=0.50 robust=+0.00470 meanΔAP=+0.0054 medΔAP=+0.0065 q25=+0.0005 wins=11/14 [SELECT] epoch=06/10 loss=0.09555 best_alpha=0.25 robust=+0.00418 meanΔAP=+0.0046 medΔAP=+0.0053 q25=+0.0025 wins=11/14 [VARIANT FLAT_RESIDUAL] selected_epochs=1 selected_alpha=0.75 robust=+0.008418 [REFIT] epoch=01/1 loss=0.11618 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.14109 best_alpha=0.25 robust=+0.00212 meanΔAP=+0.0038 medΔAP=+0.0015 q25=-0.0013 wins=8/14 [SELECT] epoch=02/10 loss=0.12847 best_alpha=0.75 robust=+0.00849 meanΔAP=+0.0090 medΔAP=+0.0116 q25=+0.0039 wins=11/14 [SELECT] epoch=03/10 loss=0.12397 best_alpha=0.25 robust=+0.00345 meanΔAP=+0.0039 medΔAP=+0.0036 q25=+0.0024 wins=12/14 [SELECT] epoch=04/10 loss=0.11828 best_alpha=0.50 robust=+0.00715 meanΔAP=+0.0069 medΔAP=+0.0107 q25=+0.0035 wins=11/14 [SELECT] epoch=05/10 loss=0.11578 best_alpha=0.25 robust=+0.00310 meanΔAP=+0.0033 medΔAP=+0.0046 q25=+0.0007 wins=11/14 [SELECT] epoch=06/10 loss=0.11455 best_alpha=0.25 robust=+0.00343 meanΔAP=+0.0035 medΔAP=+0.0050 q25=+0.0011 wins=11/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=2 selected_alpha=0.75 robust=+0.008487 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/2 loss=0.13783 [REFIT] epoch=02/2 loss=0.13154 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.13942 best_alpha=1.00 robust=+0.01082 meanΔAP=+0.0138 medΔAP=+0.0139 q25=+0.0003 wins=11/14 [SELECT] epoch=02/10 loss=0.13056 best_alpha=0.50 robust=+0.00636 meanΔAP=+0.0065 medΔAP=+0.0093 q25=+0.0022 wins=12/14 [SELECT] epoch=03/10 loss=0.12195 best_alpha=0.25 robust=+0.00681 meanΔAP=+0.0076 medΔAP=+0.0089 q25=+0.0025 wins=12/14 [SELECT] epoch=04/10 loss=0.11902 best_alpha=0.50 robust=+0.00798 meanΔAP=+0.0075 medΔAP=+0.0141 q25=-0.0020 wins=10/14 [SELECT] epoch=05/10 loss=0.11689 best_alpha=0.25 robust=+0.00470 meanΔAP=+0.0047 medΔAP=+0.0066 q25=+0.0020 wins=12/14 [SELECT] epoch=06/10 loss=0.11587 best_alpha=0.50 robust=+0.01063 meanΔAP=+0.0108 medΔAP=+0.0142 q25=+0.0071 wins=11/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=1.00 robust=+0.010817 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.13892 SUITE21 TARGET FOLD 5 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 6398 0.0549 0.2751 0.8292 0.0457 0.1828 0.6667 0.4359 khoa_cong_nghe_thong_tin suite21_flat_residual 6398 0.0549 0.2758 0.8300 0.0458 0.1820 0.6638 0.4341 khoa_cong_nghe_thong_tin suite21_temporal_residual 6398 0.0549 0.2789 0.8353 0.0461 0.1836 0.6695 0.4378 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 6398 0.0549 0.2795 0.8329 0.0466 0.1859 0.6781 0.4434 khoa_quan_tri_kinh_doanh_va_du_lich committee 7604 0.0535 0.3729 0.8769 0.0406 0.2005 0.7494 0.4843 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 7604 0.0535 0.3755 0.8771 0.0405 0.1999 0.7469 0.4827 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 7604 0.0535 0.3805 0.8801 0.0402 0.2012 0.7518 0.4859 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 7604 0.0535 0.3810 0.8819 0.0402 0.2045 0.7641 0.4938 khoa_quoc_te_hoc committee 2674 0.0217 0.1635 0.8346 0.0197 0.0692 0.6379 0.2412 khoa_quoc_te_hoc suite21_flat_residual 2674 0.0217 0.1654 0.8393 0.0198 0.0729 0.6724 0.2542 khoa_quoc_te_hoc suite21_temporal_residual 2674 0.0217 0.1706 0.8430 0.0198 0.0729 0.6724 0.2542 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2674 0.0217 0.1827 0.8386 0.0204 0.0748 0.6897 0.2608 khoa_tieng_anh committee 8345 0.0158 0.1531 0.8330 0.0146 0.0497 0.6288 0.1889 khoa_tieng_anh suite21_flat_residual 8345 0.0158 0.1590 0.8387 0.0146 0.0509 0.6439 0.1934 khoa_tieng_anh suite21_temporal_residual 8345 0.0158 0.1457 0.8490 0.0146 0.0503 0.6364 0.1912 khoa_tieng_anh suite21_guarded_temporal_residual 8345 0.0158 0.1621 0.8465 0.0148 0.0497 0.6288 0.1889 khoa_tieng_bo_dao_nha committee 1138 0.0703 0.3129 0.8595 0.0556 0.2456 0.7000 0.5109 khoa_tieng_bo_dao_nha suite21_flat_residual 1138 0.0703 0.3290 0.8603 0.0555 0.2500 0.7125 0.5201 khoa_tieng_bo_dao_nha suite21_temporal_residual 1138 0.0703 0.3103 0.8699 0.0575 0.2588 0.7375 0.5383 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 1138 0.0703 0.3183 0.8610 0.0593 0.2368 0.6750 0.4927 khoa_tieng_duc committee 1975 0.0299 0.1602 0.8684 0.0270 0.1114 0.7458 0.3487 khoa_tieng_duc suite21_flat_residual 1975 0.0299 0.1876 0.8710 0.0267 0.1063 0.7119 0.3328 khoa_tieng_duc suite21_temporal_residual 1975 0.0299 0.2009 0.8687 0.0269 0.1139 0.7627 0.3566 khoa_tieng_duc suite21_guarded_temporal_residual 1975 0.0299 0.2132 0.8786 0.0267 0.1215 0.8136 0.3803 khoa_tieng_han_quoc committee 3573 0.0188 0.1911 0.8916 0.0167 0.0783 0.8358 0.2848 khoa_tieng_han_quoc suite21_flat_residual 3573 0.0188 0.1858 0.8983 0.0166 0.0783 0.8358 0.2848 khoa_tieng_han_quoc suite21_temporal_residual 3573 0.0188 0.1777 0.9008 0.0167 0.0797 0.8507 0.2899 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3573 0.0188 0.1822 0.8990 0.0167 0.0797 0.8507 0.2899 khoa_tieng_italia committee 2129 0.0390 0.2155 0.7906 0.0346 0.1244 0.6386 0.3496 khoa_tieng_italia suite21_flat_residual 2129 0.0390 0.2069 0.7905 0.0349 0.1197 0.6145 0.3364 khoa_tieng_italia suite21_temporal_residual 2129 0.0390 0.1639 0.7934 0.0360 0.1221 0.6265 0.3430 khoa_tieng_italia suite21_guarded_temporal_residual 2129 0.0390 0.1660 0.7864 0.0367 0.1150 0.5904 0.3232 khoa_tieng_nga committee 2047 0.0655 0.4660 0.9261 0.0458 0.2910 0.8881 0.6296 khoa_tieng_nga suite21_flat_residual 2047 0.0655 0.4708 0.9265 0.0458 0.2910 0.8881 0.6296 khoa_tieng_nga suite21_temporal_residual 2047 0.0655 0.4694 0.9277 0.0471 0.2861 0.8731 0.6190 khoa_tieng_nga suite21_guarded_temporal_residual 2047 0.0655 0.4757 0.9257 0.0485 0.2812 0.8582 0.6085 khoa_tieng_nhat committee 2593 0.0266 0.2264 0.8362 0.0232 0.1021 0.7681 0.3333 khoa_tieng_nhat suite21_flat_residual 2593 0.0266 0.2301 0.8450 0.0231 0.1021 0.7681 0.3333 khoa_tieng_nhat suite21_temporal_residual 2593 0.0266 0.2509 0.8462 0.0228 0.1021 0.7681 0.3333 khoa_tieng_nhat suite21_guarded_temporal_residual 2593 0.0266 0.2331 0.8468 0.0234 0.1021 0.7681 0.3333 khoa_tieng_phap committee 2983 0.0191 0.2549 0.8676 0.0162 0.0754 0.7895 0.2727 khoa_tieng_phap suite21_flat_residual 2983 0.0191 0.2509 0.8703 0.0162 0.0754 0.7895 0.2727 khoa_tieng_phap suite21_temporal_residual 2983 0.0191 0.2239 0.8724 0.0169 0.0771 0.8070 0.2788 khoa_tieng_phap suite21_guarded_temporal_residual 2983 0.0191 0.2350 0.8745 0.0169 0.0754 0.7895 0.2727 khoa_tieng_tay_ban_nha committee 1606 0.0523 0.2652 0.8297 0.0444 0.1869 0.7143 0.4566 khoa_tieng_tay_ban_nha suite21_flat_residual 1606 0.0523 0.2794 0.8326 0.0434 0.1869 0.7143 0.4566 khoa_tieng_tay_ban_nha suite21_temporal_residual 1606 0.0523 0.2752 0.8352 0.0439 0.1838 0.7024 0.4490 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1606 0.0523 0.3045 0.8394 0.0427 0.1838 0.7024 0.4490 khoa_tieng_trung_quoc committee 6995 0.0222 0.3319 0.9355 0.0180 0.1008 0.9097 0.3492 khoa_tieng_trung_quoc suite21_flat_residual 6995 0.0222 0.3358 0.9366 0.0180 0.1022 0.9226 0.3541 khoa_tieng_trung_quoc suite21_temporal_residual 6995 0.0222 0.3481 0.9399 0.0178 0.1044 0.9419 0.3616 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 6995 0.0222 0.3464 0.9419 0.0180 0.1058 0.9548 0.3665 khoa_viet_nam_hoc committee 638 0.0956 0.5245 0.9075 0.0599 0.3828 0.8033 0.6586 khoa_viet_nam_hoc suite21_flat_residual 638 0.0956 0.5437 0.9175 0.0585 0.3906 0.8197 0.6720 khoa_viet_nam_hoc suite21_temporal_residual 638 0.0956 0.5595 0.9218 0.0631 0.3828 0.8033 0.6586 khoa_viet_nam_hoc suite21_guarded_temporal_residual 638 0.0956 0.5547 0.9195 0.0636 0.3828 0.8033 0.6586 SUITE21 TARGET FOLD 5 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.2751 0.2795 0.0045 0.0134 0.9844 0.9109 0.0367 0.0098 khoa_quan_tri_kinh_doanh_va_du_lich 0.3729 0.3810 0.0081 0.0114 0.9895 0.9257 0.0379 0.0089 khoa_quoc_te_hoc 0.1635 0.1827 0.0193 0.0097 0.9831 0.8991 0.0367 0.0085 khoa_tieng_anh 0.1531 0.1621 0.0090 0.0043 0.9932 0.9047 0.0227 0.0004 khoa_tieng_bo_dao_nha 0.3129 0.3183 0.0054 0.0264 0.9916 0.9211 0.0941 0.0167 khoa_tieng_duc 0.1602 0.2132 0.0530 0.0120 0.9780 0.8481 0.0622 0.0060 khoa_tieng_han_quoc 0.1911 0.1822 -0.0089 0.0029 0.9937 0.9594 0.0071 -0.0012 khoa_tieng_italia 0.2155 0.1660 -0.0495 0.0106 0.9865 0.8897 0.0193 0.0063 khoa_tieng_nga 0.4660 0.4757 0.0097 0.0222 0.9888 0.9144 0.0729 0.0158 khoa_tieng_nhat 0.2264 0.2331 0.0066 0.0067 0.9934 0.9480 0.0288 0.0010 khoa_tieng_phap 0.2549 0.2350 -0.0198 0.0062 0.9899 0.9196 0.0411 0.0032 khoa_tieng_tay_ban_nha 0.2652 0.3045 0.0393 0.0108 0.9833 0.9315 0.0518 0.0042 khoa_tieng_trung_quoc 0.3319 0.3464 0.0145 0.0068 0.9900 0.9457 0.0433 -0.0004 khoa_viet_nam_hoc 0.5245 0.5547 0.0303 0.0409 0.9897 0.8906 0.1160 0.0267 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 6 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=470,165 failures=10,533 (2.24%) target=43,359 failures=2,395 (5.52%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 5651 0.0996 0.0000 5.2938 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 7442 0.0797 0.0000 5.2790 6.0000 khoa_quoc_te_hoc 2466 0.0243 0.0000 5.3114 6.0000 khoa_tieng_anh 6204 0.0197 0.0000 4.8040 5.0000 khoa_tieng_bo_dao_nha 892 0.0818 0.0000 4.9821 5.0000 khoa_tieng_duc 1839 0.0457 0.0000 4.8825 5.0000 khoa_tieng_han_quoc 3277 0.0375 0.0000 4.9823 5.0000 khoa_tieng_italia 2026 0.0464 0.0000 4.8860 5.0000 khoa_tieng_nga 1842 0.1113 0.0000 4.6976 5.0000 khoa_tieng_nhat 2530 0.0348 0.0000 4.9166 5.0000 khoa_tieng_phap 2263 0.0221 0.0000 4.5409 5.0000 khoa_tieng_tay_ban_nha 1333 0.0638 0.0000 4.5236 5.0000 khoa_tieng_trung_quoc 4990 0.0335 0.0000 4.7257 5.0000 khoa_viet_nam_hoc 604 0.1457 0.0000 4.2533 3.0000 [SPLIT] train=358,455 validation=111,710 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11834 best_alpha=0.75 robust=+0.00885 meanΔAP=+0.0110 medΔAP=+0.0092 q25=+0.0074 wins=13/14 [SELECT] epoch=02/10 loss=0.11034 best_alpha=0.50 robust=+0.00625 meanΔAP=+0.0080 medΔAP=+0.0071 q25=+0.0022 wins=13/14 [SELECT] epoch=03/10 loss=0.10506 best_alpha=0.50 robust=+0.00513 meanΔAP=+0.0055 medΔAP=+0.0060 q25=+0.0039 wins=11/14 [SELECT] epoch=04/10 loss=0.10149 best_alpha=0.50 robust=+0.00401 meanΔAP=+0.0046 medΔAP=+0.0059 q25=-0.0024 wins=9/14 [SELECT] epoch=05/10 loss=0.09811 best_alpha=0.25 robust=+0.00296 meanΔAP=+0.0044 medΔAP=+0.0027 q25=+0.0014 wins=12/14 [SELECT] epoch=06/10 loss=0.09690 best_alpha=0.50 robust=+0.00335 meanΔAP=+0.0044 medΔAP=+0.0034 q25=-0.0006 wins=9/14 [VARIANT FLAT_RESIDUAL] selected_epochs=1 selected_alpha=0.75 robust=+0.008849 [REFIT] epoch=01/1 loss=0.12016 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.14144 best_alpha=0.50 robust=+0.00672 meanΔAP=+0.0087 medΔAP=+0.0076 q25=+0.0009 wins=11/14 [SELECT] epoch=02/10 loss=0.13000 best_alpha=0.50 robust=+0.00953 meanΔAP=+0.0120 medΔAP=+0.0112 q25=+0.0027 wins=12/14 [SELECT] epoch=03/10 loss=0.12332 best_alpha=0.50 robust=+0.00680 meanΔAP=+0.0088 medΔAP=+0.0065 q25=+0.0027 wins=11/14 [SELECT] epoch=04/10 loss=0.11877 best_alpha=0.50 robust=+0.00782 meanΔAP=+0.0086 medΔAP=+0.0100 q25=+0.0032 wins=12/14 [SELECT] epoch=05/10 loss=0.11594 best_alpha=0.25 robust=+0.00617 meanΔAP=+0.0067 medΔAP=+0.0073 q25=+0.0029 wins=14/14 [SELECT] epoch=06/10 loss=0.11339 best_alpha=0.50 robust=+0.00854 meanΔAP=+0.0095 medΔAP=+0.0094 q25=+0.0063 wins=12/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=2 selected_alpha=0.50 robust=+0.009527 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/2 loss=0.14857 [REFIT] epoch=02/2 loss=0.14083 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.13994 best_alpha=0.75 robust=+0.00840 meanΔAP=+0.0105 medΔAP=+0.0073 q25=+0.0037 wins=12/14 [SELECT] epoch=02/10 loss=0.12941 best_alpha=0.50 robust=+0.01053 meanΔAP=+0.0134 medΔAP=+0.0102 q25=+0.0053 wins=13/14 [SELECT] epoch=03/10 loss=0.12395 best_alpha=0.50 robust=+0.00681 meanΔAP=+0.0087 medΔAP=+0.0064 q25=+0.0033 wins=12/14 [SELECT] epoch=04/10 loss=0.11996 best_alpha=0.75 robust=+0.01099 meanΔAP=+0.0137 medΔAP=+0.0102 q25=+0.0073 wins=13/14 [SELECT] epoch=05/10 loss=0.11705 best_alpha=0.50 robust=+0.01455 meanΔAP=+0.0175 medΔAP=+0.0156 q25=+0.0094 wins=14/14 [SELECT] epoch=06/10 loss=0.11741 best_alpha=0.75 robust=+0.01377 meanΔAP=+0.0180 medΔAP=+0.0125 q25=+0.0067 wins=14/14 [SELECT] epoch=07/10 loss=0.11485 best_alpha=0.50 robust=+0.01415 meanΔAP=+0.0181 medΔAP=+0.0151 q25=+0.0059 wins=14/14 [SELECT] epoch=08/10 loss=0.11479 best_alpha=0.50 robust=+0.00743 meanΔAP=+0.0093 medΔAP=+0.0079 q25=+0.0027 wins=12/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=5 selected_alpha=0.50 robust=+0.014550 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/5 loss=0.14801 [REFIT] epoch=05/5 loss=0.12889 SUITE21 TARGET FOLD 6 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 5651 0.0996 0.3724 0.8361 0.0752 0.3195 0.6412 0.5337 khoa_cong_nghe_thong_tin suite21_flat_residual 5651 0.0996 0.3742 0.8366 0.0751 0.3177 0.6377 0.5308 khoa_cong_nghe_thong_tin suite21_temporal_residual 5651 0.0996 0.3820 0.8405 0.0742 0.3248 0.6519 0.5426 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 5651 0.0996 0.3805 0.8398 0.0746 0.3204 0.6430 0.5352 khoa_quan_tri_kinh_doanh_va_du_lich committee 7442 0.0797 0.3777 0.8583 0.0606 0.2782 0.6981 0.5363 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 7442 0.0797 0.3789 0.8593 0.0606 0.2796 0.7015 0.5389 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 7442 0.0797 0.3896 0.8640 0.0597 0.2870 0.7201 0.5531 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 7442 0.0797 0.3822 0.8609 0.0604 0.2796 0.7015 0.5389 khoa_quoc_te_hoc committee 2466 0.0243 0.2778 0.8239 0.0215 0.0811 0.6667 0.2729 khoa_quoc_te_hoc suite21_flat_residual 2466 0.0243 0.2765 0.8260 0.0214 0.0832 0.6833 0.2797 khoa_quoc_te_hoc suite21_temporal_residual 2466 0.0243 0.2829 0.8259 0.0214 0.0852 0.7000 0.2865 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2466 0.0243 0.2754 0.8260 0.0212 0.0811 0.6667 0.2729 khoa_tieng_anh committee 6204 0.0197 0.2882 0.8256 0.0175 0.0620 0.6311 0.2227 khoa_tieng_anh suite21_flat_residual 6204 0.0197 0.2908 0.8319 0.0173 0.0637 0.6475 0.2285 khoa_tieng_anh suite21_temporal_residual 6204 0.0197 0.2909 0.8275 0.0174 0.0620 0.6311 0.2227 khoa_tieng_anh suite21_guarded_temporal_residual 6204 0.0197 0.2903 0.8301 0.0175 0.0629 0.6393 0.2256 khoa_tieng_bo_dao_nha committee 892 0.0818 0.5210 0.8806 0.0550 0.2865 0.6986 0.5426 khoa_tieng_bo_dao_nha suite21_flat_residual 892 0.0818 0.5228 0.8809 0.0550 0.2921 0.7123 0.5532 khoa_tieng_bo_dao_nha suite21_temporal_residual 892 0.0818 0.5209 0.8808 0.0550 0.2865 0.6986 0.5426 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 892 0.0818 0.5200 0.8792 0.0556 0.2921 0.7123 0.5532 khoa_tieng_duc committee 1839 0.0457 0.4558 0.9230 0.0334 0.1984 0.8690 0.5185 khoa_tieng_duc suite21_flat_residual 1839 0.0457 0.4618 0.9249 0.0331 0.2011 0.8810 0.5256 khoa_tieng_duc suite21_temporal_residual 1839 0.0457 0.4580 0.9242 0.0333 0.1984 0.8690 0.5185 khoa_tieng_duc suite21_guarded_temporal_residual 1839 0.0457 0.4626 0.9279 0.0329 0.2011 0.8810 0.5256 khoa_tieng_han_quoc committee 3277 0.0375 0.2531 0.8587 0.0323 0.1435 0.7642 0.4098 khoa_tieng_han_quoc suite21_flat_residual 3277 0.0375 0.2609 0.8617 0.0321 0.1466 0.7805 0.4185 khoa_tieng_han_quoc suite21_temporal_residual 3277 0.0375 0.2567 0.8595 0.0322 0.1420 0.7561 0.4054 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3277 0.0375 0.2630 0.8606 0.0320 0.1405 0.7480 0.4010 khoa_tieng_italia committee 2026 0.0464 0.2286 0.7987 0.0467 0.1481 0.6383 0.3841 khoa_tieng_italia suite21_flat_residual 2026 0.0464 0.2295 0.7966 0.0467 0.1506 0.6489 0.3905 khoa_tieng_italia suite21_temporal_residual 2026 0.0464 0.2288 0.7969 0.0466 0.1481 0.6383 0.3841 khoa_tieng_italia suite21_guarded_temporal_residual 2026 0.0464 0.2308 0.7980 0.0466 0.1481 0.6383 0.3841 khoa_tieng_nga committee 1842 0.1113 0.4801 0.7788 0.0777 0.3179 0.5707 0.4924 khoa_tieng_nga suite21_flat_residual 1842 0.1113 0.4820 0.7822 0.0776 0.3234 0.5805 0.5008 khoa_tieng_nga suite21_temporal_residual 1842 0.1113 0.4809 0.7806 0.0777 0.3207 0.5756 0.4966 khoa_tieng_nga suite21_guarded_temporal_residual 1842 0.1113 0.4812 0.7799 0.0775 0.3179 0.5707 0.4924 khoa_tieng_nhat committee 2530 0.0348 0.1763 0.8325 0.0328 0.1126 0.6477 0.3322 khoa_tieng_nhat suite21_flat_residual 2530 0.0348 0.1797 0.8384 0.0326 0.1166 0.6705 0.3438 khoa_tieng_nhat suite21_temporal_residual 2530 0.0348 0.1800 0.8380 0.0326 0.1186 0.6818 0.3497 khoa_tieng_nhat suite21_guarded_temporal_residual 2530 0.0348 0.1847 0.8435 0.0323 0.1146 0.6591 0.3380 khoa_tieng_phap committee 2263 0.0221 0.2791 0.9146 0.0199 0.0949 0.8600 0.3292 khoa_tieng_phap suite21_flat_residual 2263 0.0221 0.2809 0.9153 0.0199 0.0949 0.8600 0.3292 khoa_tieng_phap suite21_temporal_residual 2263 0.0221 0.2808 0.9170 0.0199 0.0949 0.8600 0.3292 khoa_tieng_phap suite21_guarded_temporal_residual 2263 0.0221 0.2803 0.9171 0.0198 0.0971 0.8800 0.3369 khoa_tieng_tay_ban_nha committee 1333 0.0638 0.3718 0.8478 0.0511 0.2285 0.7176 0.5025 khoa_tieng_tay_ban_nha suite21_flat_residual 1333 0.0638 0.3781 0.8556 0.0509 0.2285 0.7176 0.5025 khoa_tieng_tay_ban_nha suite21_temporal_residual 1333 0.0638 0.3738 0.8537 0.0510 0.2285 0.7176 0.5025 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1333 0.0638 0.3824 0.8607 0.0502 0.2285 0.7176 0.5025 khoa_tieng_trung_quoc committee 4990 0.0335 0.3394 0.8835 0.0278 0.1333 0.7964 0.3992 khoa_tieng_trung_quoc suite21_flat_residual 4990 0.0335 0.3457 0.8896 0.0276 0.1333 0.7964 0.3992 khoa_tieng_trung_quoc suite21_temporal_residual 4990 0.0335 0.3440 0.8899 0.0277 0.1333 0.7964 0.3992 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 4990 0.0335 0.3587 0.8973 0.0274 0.1333 0.7964 0.3992 khoa_viet_nam_hoc committee 604 0.1457 0.6504 0.9018 0.0801 0.5702 0.7841 0.7294 khoa_viet_nam_hoc suite21_flat_residual 604 0.1457 0.6500 0.9009 0.0810 0.5620 0.7727 0.7188 khoa_viet_nam_hoc suite21_temporal_residual 604 0.1457 0.6510 0.9020 0.0803 0.5620 0.7727 0.7188 khoa_viet_nam_hoc suite21_guarded_temporal_residual 604 0.1457 0.6520 0.9046 0.0804 0.5537 0.7614 0.7082 SUITE21 TARGET FOLD 6 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.3724 0.3805 0.0081 0.0024 0.9986 0.9761 0.0039 -0.0001 khoa_quan_tri_kinh_doanh_va_du_lich 0.3777 0.3822 0.0046 0.0015 0.9992 0.9785 0.0004 -0.0009 khoa_quoc_te_hoc 0.2778 0.2754 -0.0025 0.0022 0.9988 0.9777 -0.0037 -0.0013 khoa_tieng_anh 0.2882 0.2903 0.0021 0.0026 0.9981 0.9605 0.0013 -0.0003 khoa_tieng_bo_dao_nha 0.5210 0.5200 -0.0010 0.0059 0.9966 0.9494 0.0057 0.0023 khoa_tieng_duc 0.4558 0.4626 0.0067 0.0030 0.9931 0.9348 0.0048 -0.0003 khoa_tieng_han_quoc 0.2531 0.2630 0.0099 0.0025 0.9978 0.9588 0.0025 -0.0006 khoa_tieng_italia 0.2286 0.2308 0.0022 0.0044 0.9821 0.9654 0.0037 0.0014 khoa_tieng_nga 0.4801 0.4812 0.0011 0.0018 0.9977 0.9755 0.0008 0.0000 khoa_tieng_nhat 0.1763 0.1847 0.0085 0.0032 0.9932 0.9269 0.0081 0.0004 khoa_tieng_phap 0.2791 0.2803 0.0012 0.0015 0.9948 0.9294 -0.0000 0.0001 khoa_tieng_tay_ban_nha 0.3718 0.3824 0.0106 0.0041 0.9936 0.9775 0.0068 -0.0006 khoa_tieng_trung_quoc 0.3394 0.3587 0.0194 0.0029 0.9942 0.9489 0.0166 0.0013 khoa_viet_nam_hoc 0.6504 0.6520 0.0016 0.0076 0.9953 0.9752 0.0060 0.0070 ================================================================================================================================================================ SUITE21 — COMMITTEE-ANCHORED GUARDED TEMPORAL RESIDUAL | TARGET FOLD 7 ================================================================================================================================================================ [DATA] faculties=14 common=136 committee_members=3 history=514,858 failures=13,041 (2.53%) target=53,645 failures=4,179 (7.79%) HISTORY COVERAGE — TARGET -------------------------------------------------------------------------------- faculty rows prevalence history_zero_rate history_mean_semesters history_median_semesters khoa_cong_nghe_thong_tin 6926 0.1513 0.0000 5.0739 6.0000 khoa_quan_tri_kinh_doanh_va_du_lich 8989 0.1249 0.0000 4.9774 6.0000 khoa_quoc_te_hoc 2755 0.0465 0.0000 5.1517 6.0000 khoa_tieng_anh 8369 0.0299 0.0000 4.2629 4.0000 khoa_tieng_bo_dao_nha 1209 0.1183 0.0000 4.0232 4.0000 khoa_tieng_duc 2284 0.0486 0.0000 4.3424 4.0000 khoa_tieng_han_quoc 3434 0.0364 0.0000 4.4793 4.0000 khoa_tieng_italia 2453 0.1035 0.0000 4.4317 4.0000 khoa_tieng_nga 1924 0.0800 0.0000 4.1112 4.0000 khoa_tieng_nhat 2795 0.0519 0.0000 4.1395 4.0000 khoa_tieng_phap 2915 0.0408 0.0000 4.1825 4.0000 khoa_tieng_tay_ban_nha 1957 0.0971 0.0000 4.1114 4.0000 khoa_tieng_trung_quoc 6657 0.0326 0.0000 4.4822 4.0000 khoa_viet_nam_hoc 978 0.1759 0.0000 3.6564 4.0000 [SPLIT] train=405,366 validation=109,492 val_blocks=3 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT FLAT_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.11544 best_alpha=1.00 robust=+0.00705 meanΔAP=+0.0088 medΔAP=+0.0075 q25=+0.0016 wins=11/14 [SELECT] epoch=02/10 loss=0.11353 best_alpha=0.75 robust=+0.00599 meanΔAP=+0.0072 medΔAP=+0.0074 q25=+0.0003 wins=11/14 [SELECT] epoch=03/10 loss=0.10798 best_alpha=0.50 robust=+0.00234 meanΔAP=+0.0034 medΔAP=+0.0015 q25=-0.0007 wins=8/14 [SELECT] epoch=04/10 loss=0.10476 best_alpha=0.75 robust=+0.00561 meanΔAP=+0.0062 medΔAP=+0.0063 q25=+0.0007 wins=12/14 [SELECT] epoch=05/10 loss=0.10421 best_alpha=0.50 robust=+0.00294 meanΔAP=+0.0030 medΔAP=+0.0029 q25=+0.0008 wins=12/14 [SELECT] epoch=06/10 loss=0.10152 best_alpha=0.50 robust=+0.00396 meanΔAP=+0.0048 medΔAP=+0.0032 q25=+0.0012 wins=11/14 [VARIANT FLAT_RESIDUAL] selected_epochs=1 selected_alpha=1.00 robust=+0.007049 [REFIT] epoch=01/1 loss=0.12758 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT TEMPORAL_RESIDUAL] residual learning with Committee identity fallback /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [SELECT] epoch=01/10 loss=0.14209 best_alpha=1.00 robust=+0.00926 meanΔAP=+0.0118 medΔAP=+0.0087 q25=+0.0032 wins=12/14 [SELECT] epoch=02/10 loss=0.13265 best_alpha=1.00 robust=+0.00531 meanΔAP=+0.0065 medΔAP=+0.0055 q25=+0.0003 wins=10/14 [SELECT] epoch=03/10 loss=0.12730 best_alpha=0.50 robust=+0.00354 meanΔAP=+0.0033 medΔAP=+0.0051 q25=-0.0004 wins=9/14 [SELECT] epoch=04/10 loss=0.12487 best_alpha=0.75 robust=+0.00453 meanΔAP=+0.0053 medΔAP=+0.0044 q25=+0.0015 wins=11/14 [SELECT] epoch=05/10 loss=0.12215 best_alpha=0.50 robust=+0.00616 meanΔAP=+0.0059 medΔAP=+0.0081 q25=+0.0020 wins=11/14 [SELECT] epoch=06/10 loss=0.12119 best_alpha=0.50 robust=+0.00506 meanΔAP=+0.0050 medΔAP=+0.0065 q25=-0.0004 wins=10/14 [VARIANT TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=1.00 robust=+0.009262 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.15654 ----------------------------------------------------------------------------------------------------------------------------- [VARIANT GUARDED_TEMPORAL_RESIDUAL] residual learning with Committee identity fallback [SELECT] epoch=01/10 loss=0.14093 best_alpha=1.00 robust=+0.00849 meanΔAP=+0.0107 medΔAP=+0.0084 q25=+0.0033 wins=13/14 [SELECT] epoch=02/10 loss=0.13164 best_alpha=0.50 robust=+0.00433 meanΔAP=+0.0052 medΔAP=+0.0043 q25=+0.0025 wins=12/14 [SELECT] epoch=03/10 loss=0.12789 best_alpha=1.00 robust=+0.00649 meanΔAP=+0.0078 medΔAP=+0.0064 q25=+0.0021 wins=13/14 [SELECT] epoch=04/10 loss=0.12383 best_alpha=0.75 robust=+0.00558 meanΔAP=+0.0069 medΔAP=+0.0061 q25=+0.0012 wins=12/14 [SELECT] epoch=05/10 loss=0.12362 best_alpha=0.75 robust=+0.00605 meanΔAP=+0.0058 medΔAP=+0.0085 q25=+0.0011 wins=11/14 [SELECT] epoch=06/10 loss=0.12056 best_alpha=0.75 robust=+0.00567 meanΔAP=+0.0066 medΔAP=+0.0069 q25=-0.0014 wins=10/14 [VARIANT GUARDED_TEMPORAL_RESIDUAL] selected_epochs=1 selected_alpha=1.00 robust=+0.008491 /root/miniconda3/envs/Huy/lib/python3.13/site-packages/torch/nn/modules/transformer.py:382: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True warnings.warn( [REFIT] epoch=01/1 loss=0.15792 SUITE21 TARGET FOLD 7 — NEW MODEL RESULTS -------------------------------------------------------------------------------- faculty model rows prevalence ap auc brier precision_at_20 recall_at_20 fbeta_at_20 khoa_cong_nghe_thong_tin committee 6926 0.1513 0.4527 0.8086 0.1098 0.4217 0.5573 0.5236 khoa_cong_nghe_thong_tin suite21_flat_residual 6926 0.1513 0.4569 0.8143 0.1079 0.4303 0.5687 0.5343 khoa_cong_nghe_thong_tin suite21_temporal_residual 6926 0.1513 0.4481 0.8041 0.1063 0.4188 0.5534 0.5200 khoa_cong_nghe_thong_tin suite21_guarded_temporal_residual 6926 0.1513 0.4578 0.8055 0.1064 0.4260 0.5630 0.5290 khoa_quan_tri_kinh_doanh_va_du_lich committee 8989 0.1249 0.4207 0.8205 0.0925 0.3643 0.5833 0.5207 khoa_quan_tri_kinh_doanh_va_du_lich suite21_flat_residual 8989 0.1249 0.4206 0.8231 0.0915 0.3648 0.5841 0.5215 khoa_quan_tri_kinh_doanh_va_du_lich suite21_temporal_residual 8989 0.1249 0.4152 0.8165 0.0906 0.3498 0.5601 0.5000 khoa_quan_tri_kinh_doanh_va_du_lich suite21_guarded_temporal_residual 8989 0.1249 0.4093 0.8158 0.0914 0.3493 0.5592 0.4992 khoa_quoc_te_hoc committee 2755 0.0465 0.2446 0.8356 0.0412 0.1488 0.6406 0.3857 khoa_quoc_te_hoc suite21_flat_residual 2755 0.0465 0.2595 0.8418 0.0407 0.1525 0.6562 0.3951 khoa_quoc_te_hoc suite21_temporal_residual 2755 0.0465 0.2654 0.8413 0.0403 0.1543 0.6641 0.3998 khoa_quoc_te_hoc suite21_guarded_temporal_residual 2755 0.0465 0.2477 0.8266 0.0406 0.1434 0.6172 0.3716 khoa_tieng_anh committee 8369 0.0299 0.2209 0.8387 0.0259 0.1010 0.6760 0.3160 khoa_tieng_anh suite21_flat_residual 8369 0.0299 0.2229 0.8451 0.0258 0.1027 0.6880 0.3216 khoa_tieng_anh suite21_temporal_residual 8369 0.0299 0.2238 0.8464 0.0258 0.1033 0.6920 0.3235 khoa_tieng_anh suite21_guarded_temporal_residual 8369 0.0299 0.2259 0.8417 0.0257 0.1010 0.6760 0.3160 khoa_tieng_bo_dao_nha committee 1209 0.1183 0.4075 0.7939 0.0870 0.3264 0.5524 0.4853 khoa_tieng_bo_dao_nha suite21_flat_residual 1209 0.1183 0.4124 0.7960 0.0867 0.3223 0.5455 0.4791 khoa_tieng_bo_dao_nha suite21_temporal_residual 1209 0.1183 0.3968 0.7964 0.0874 0.3306 0.5594 0.4914 khoa_tieng_bo_dao_nha suite21_guarded_temporal_residual 1209 0.1183 0.4220 0.7941 0.0868 0.3264 0.5524 0.4853 khoa_tieng_duc committee 2284 0.0486 0.2956 0.8739 0.0396 0.1860 0.7658 0.4717 khoa_tieng_duc suite21_flat_residual 2284 0.0486 0.3073 0.8795 0.0392 0.1882 0.7748 0.4772 khoa_tieng_duc suite21_temporal_residual 2284 0.0486 0.2962 0.8783 0.0396 0.1860 0.7658 0.4717 khoa_tieng_duc suite21_guarded_temporal_residual 2284 0.0486 0.3094 0.8758 0.0392 0.1904 0.7838 0.4828 khoa_tieng_han_quoc committee 3434 0.0364 0.3786 0.8930 0.0279 0.1456 0.8000 0.4212 khoa_tieng_han_quoc suite21_flat_residual 3434 0.0364 0.3929 0.8988 0.0274 0.1426 0.7840 0.4128 khoa_tieng_han_quoc suite21_temporal_residual 3434 0.0364 0.4285 0.8995 0.0264 0.1426 0.7840 0.4128 khoa_tieng_han_quoc suite21_guarded_temporal_residual 3434 0.0364 0.4093 0.9044 0.0271 0.1514 0.8320 0.4381 khoa_tieng_italia committee 2453 0.1035 0.3767 0.8338 0.0814 0.3096 0.5984 0.5043 khoa_tieng_italia suite21_flat_residual 2453 0.1035 0.3783 0.8342 0.0804 0.3075 0.5945 0.5010 khoa_tieng_italia suite21_temporal_residual 2453 0.1035 0.3544 0.8304 0.0815 0.3096 0.5984 0.5043 khoa_tieng_italia suite21_guarded_temporal_residual 2453 0.1035 0.3805 0.8334 0.0810 0.3116 0.6024 0.5076 khoa_tieng_nga committee 1924 0.0800 0.2739 0.7723 0.0697 0.2312 0.5779 0.4446 khoa_tieng_nga suite21_flat_residual 1924 0.0800 0.2774 0.7840 0.0698 0.2338 0.5844 0.4496 khoa_tieng_nga suite21_temporal_residual 1924 0.0800 0.2769 0.7819 0.0701 0.2416 0.6039 0.4645 khoa_tieng_nga suite21_guarded_temporal_residual 1924 0.0800 0.2745 0.7835 0.0707 0.2312 0.5779 0.4446 khoa_tieng_nhat committee 2795 0.0519 0.2195 0.8187 0.0447 0.1699 0.6552 0.4170 khoa_tieng_nhat suite21_flat_residual 2795 0.0519 0.2256 0.8243 0.0445 0.1664 0.6414 0.4083 khoa_tieng_nhat suite21_temporal_residual 2795 0.0519 0.2427 0.8296 0.0441 0.1753 0.6759 0.4302 khoa_tieng_nhat suite21_guarded_temporal_residual 2795 0.0519 0.2386 0.8244 0.0446 0.1682 0.6483 0.4126 khoa_tieng_phap committee 2915 0.0408 0.2748 0.8276 0.0338 0.1235 0.6050 0.3399 khoa_tieng_phap suite21_flat_residual 2915 0.0408 0.2790 0.8371 0.0338 0.1252 0.6134 0.3447 khoa_tieng_phap suite21_temporal_residual 2915 0.0408 0.2829 0.8370 0.0336 0.1235 0.6050 0.3399 khoa_tieng_phap suite21_guarded_temporal_residual 2915 0.0408 0.2717 0.8203 0.0340 0.1149 0.5630 0.3163 khoa_tieng_tay_ban_nha committee 1957 0.0971 0.4033 0.8603 0.0715 0.3350 0.6895 0.5691 khoa_tieng_tay_ban_nha suite21_flat_residual 1957 0.0971 0.4123 0.8655 0.0701 0.3376 0.6947 0.5734 khoa_tieng_tay_ban_nha suite21_temporal_residual 1957 0.0971 0.4080 0.8664 0.0703 0.3427 0.7053 0.5821 khoa_tieng_tay_ban_nha suite21_guarded_temporal_residual 1957 0.0971 0.4355 0.8705 0.0689 0.3376 0.6947 0.5734 khoa_tieng_trung_quoc committee 6657 0.0326 0.1597 0.8197 0.0296 0.1022 0.6267 0.3092 khoa_tieng_trung_quoc suite21_flat_residual 6657 0.0326 0.1682 0.8398 0.0295 0.1119 0.6866 0.3388 khoa_tieng_trung_quoc suite21_temporal_residual 6657 0.0326 0.1625 0.8364 0.0301 0.1097 0.6728 0.3320 khoa_tieng_trung_quoc suite21_guarded_temporal_residual 6657 0.0326 0.1648 0.8287 0.0297 0.1044 0.6406 0.3161 khoa_viet_nam_hoc committee 978 0.1759 0.6040 0.8547 0.1059 0.5306 0.6047 0.5882 khoa_viet_nam_hoc suite21_flat_residual 978 0.1759 0.6045 0.8588 0.1068 0.5357 0.6105 0.5939 khoa_viet_nam_hoc suite21_temporal_residual 978 0.1759 0.5939 0.8594 0.1125 0.5408 0.6163 0.5995 khoa_viet_nam_hoc suite21_guarded_temporal_residual 978 0.1759 0.6065 0.8629 0.1122 0.5357 0.6105 0.5939 SUITE21 TARGET FOLD 7 — COMMITTEE RETENTION -------------------------------------------------------------------------------- faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta khoa_cong_nghe_thong_tin 0.4527 0.4578 0.0051 0.0212 0.9677 0.8412 0.0334 0.0161 khoa_quan_tri_kinh_doanh_va_du_lich 0.4207 0.4093 -0.0114 0.0167 0.9742 0.8454 0.0242 0.0103 khoa_quoc_te_hoc 0.2446 0.2477 0.0032 0.0061 0.9779 0.8113 0.0125 0.0032 khoa_tieng_anh 0.2209 0.2259 0.0050 0.0053 0.9938 0.9295 -0.0028 -0.0046 khoa_tieng_bo_dao_nha 0.4075 0.4220 0.0145 0.0159 0.9739 0.9380 0.0205 0.0016 khoa_tieng_duc 0.2956 0.3094 0.0138 0.0089 0.9810 0.9125 0.0132 -0.0001 khoa_tieng_han_quoc 0.3786 0.4093 0.0307 0.0067 0.9901 0.8879 0.0242 -0.0003 khoa_tieng_italia 0.3767 0.3805 0.0038 0.0065 0.9828 0.9348 0.0019 -0.0027 khoa_tieng_nga 0.2739 0.2745 0.0006 0.0156 0.9620 0.8779 0.0196 0.0038 khoa_tieng_nhat 0.2195 0.2386 0.0191 0.0118 0.9726 0.9088 0.0256 0.0020 khoa_tieng_phap 0.2748 0.2717 -0.0031 0.0038 0.9875 0.8696 -0.0007 -0.0016 khoa_tieng_tay_ban_nha 0.4033 0.4355 0.0323 0.0172 0.9778 0.8849 0.0431 0.0046 khoa_tieng_trung_quoc 0.1597 0.1648 0.0051 0.0092 0.9824 0.9031 0.0175 -0.0003 khoa_viet_nam_hoc 0.6040 0.6065 0.0025 0.0456 0.9788 0.9337 0.0551 0.0429 SUITE21 FOLD-BY-FOLD MACRO SUMMARY — ALL FACULTIES -------------------------------------------------------------------------------- fold model faculties macro_ap macro_auc macro_brier macro_ece10 macro_calibration_gap macro_f2_at_20 1 committee 14 0.2688 0.8273 0.0250 0.0124 -0.0051 0.2688 1 suite17 14 0.2876 0.8443 0.0247 0.0134 -0.0033 0.2853 1 suite18 14 0.2842 0.8406 0.0248 0.0126 -0.0033 0.2843 1 suite20_transformer 14 0.2138 0.8545 0.0492 0.0535 0.0476 0.2889 1 suite21_flat_residual 14 0.2759 0.8339 0.0249 0.0127 -0.0039 0.2752 1 suite21_guarded_temporal_residual 14 0.2702 0.8383 0.0249 0.0123 -0.0038 0.2738 1 suite21_temporal_residual 14 0.2751 0.8308 0.0251 0.0132 -0.0044 0.2741 2 committee 14 0.5042 0.9260 0.0178 0.0163 0.0040 0.3481 2 suite17 14 0.5177 0.9342 0.0182 0.0172 0.0041 0.3600 2 suite18 14 0.5175 0.9343 0.0182 0.0173 0.0041 0.3584 2 suite20_transformer 14 0.3010 0.8853 0.0988 0.1359 0.1329 0.3114 2 suite21_flat_residual 14 0.4949 0.9257 0.0185 0.0201 0.0115 0.3479 2 suite21_guarded_temporal_residual 14 0.5133 0.9284 0.0184 0.0191 0.0105 0.3508 2 suite21_temporal_residual 14 0.5011 0.9290 0.0181 0.0183 0.0090 0.3499 3 committee 14 0.2981 0.8751 0.0224 0.0264 0.0220 0.3174 3 suite17 14 0.3141 0.8952 0.0227 0.0298 0.0233 0.3308 3 suite18 14 0.3113 0.8945 0.0226 0.0290 0.0232 0.3310 3 suite20_transformer 14 0.2840 0.8772 0.0581 0.0661 0.0648 0.3234 3 suite21_flat_residual 14 0.2968 0.8777 0.0223 0.0256 0.0212 0.3196 3 suite21_guarded_temporal_residual 14 0.3112 0.8815 0.0222 0.0288 0.0231 0.3242 3 suite21_temporal_residual 14 0.3046 0.8814 0.0220 0.0239 0.0191 0.3141 4 committee 14 0.4261 0.9153 0.0213 0.0171 0.0128 0.3673 4 suite17 14 0.4360 0.9223 0.0216 0.0204 0.0150 0.3716 4 suite18 14 0.4346 0.9222 0.0216 0.0205 0.0151 0.3711 4 suite20_transformer 14 0.3453 0.9060 0.0944 0.1345 0.1345 0.3449 4 suite21_flat_residual 14 0.4298 0.9157 0.0213 0.0175 0.0135 0.3650 4 suite21_guarded_temporal_residual 14 0.4367 0.9194 0.0214 0.0173 0.0137 0.3732 4 suite21_temporal_residual 14 0.4363 0.9193 0.0214 0.0196 0.0163 0.3669 5 committee 14 0.2795 0.8633 0.0330 0.0152 0.0107 0.3960 5 suite17 14 0.2868 0.8698 0.0332 0.0184 0.0137 0.3961 5 suite18 14 0.2876 0.8703 0.0332 0.0186 0.0138 0.3983 5 suite20_transformer 14 0.2649 0.8634 0.0878 0.1201 0.1197 0.3882 5 suite21_flat_residual 14 0.2854 0.8667 0.0328 0.0157 0.0125 0.3969 5 suite21_guarded_temporal_residual 14 0.2882 0.8695 0.0339 0.0219 0.0203 0.3973 5 suite21_temporal_residual 14 0.2825 0.8702 0.0335 0.0186 0.0162 0.3998 6 committee 14 0.3623 0.8546 0.0451 0.0223 0.0043 0.4432 6 suite17 14 0.3712 0.8645 0.0445 0.0214 0.0064 0.4509 6 suite18 14 0.3708 0.8649 0.0446 0.0211 0.0063 0.4502 6 suite20_transformer 14 0.3206 0.8492 0.1018 0.1276 0.1257 0.4355 6 suite21_flat_residual 14 0.3651 0.8571 0.0451 0.0222 0.0050 0.4471 6 suite21_guarded_temporal_residual 14 0.3674 0.8590 0.0449 0.0209 0.0051 0.4438 6 suite21_temporal_residual 14 0.3657 0.8572 0.0449 0.0219 0.0048 0.4465 7 committee 14 0.3380 0.8322 0.0615 0.0258 -0.0011 0.4498 7 suite17 14 0.3445 0.8412 0.0620 0.0288 -0.0010 0.4561 7 suite18 14 0.3448 0.8409 0.0620 0.0295 -0.0012 0.4604 7 suite20_transformer 14 0.3293 0.8299 0.0981 0.1102 0.1031 0.4375 7 suite21_flat_residual 14 0.3441 0.8387 0.0610 0.0261 0.0012 0.4537 7 suite21_guarded_temporal_residual 14 0.3467 0.8348 0.0613 0.0268 0.0055 0.4490 7 suite21_temporal_residual 14 0.3425 0.8374 0.0613 0.0248 0.0034 0.4551 SUITE21 ALL-FOLD MACRO SUMMARY — F1..F7 -------------------------------------------------------------------------------- model faculties mean_fold_ap mean_fold_auc mean_fold_brier mean_fold_ece10 mean_fold_calibration_gap mean_fold_precision_at_20 mean_fold_recall_at_20 mean_fold_fbeta_at_20 council 1 0.3833 0.8557 0.0513 0.0386 0.0367 0.2198 0.7135 0.4561 suite17 14 0.3654 0.8816 0.0324 0.0213 0.0083 0.1549 0.7753 0.3787 suite18 14 0.3644 0.8811 0.0324 0.0212 0.0083 0.1552 0.7742 0.3791 suite21_guarded_temporal_residual 14 0.3620 0.8758 0.0324 0.0210 0.0106 0.1529 0.7615 0.3732 suite14 14 0.3615 0.8706 0.0409 0.0472 0.0419 0.1514 0.7484 0.3681 suite16 14 0.3601 0.8806 0.0558 0.0887 0.0879 0.1546 0.7745 0.3781 suite21_temporal_residual 14 0.3583 0.8751 0.0323 0.0200 0.0092 0.1527 0.7599 0.3724 suite11_candidate 14 0.3566 0.8715 0.0328 0.0184 0.0044 0.1522 0.7574 0.3710 suite21_flat_residual 14 0.3560 0.8736 0.0323 0.0200 0.0087 0.1527 0.7586 0.3722 committee 14 0.3539 0.8705 0.0323 0.0193 0.0068 0.1519 0.7540 0.3701 suite11_final 14 0.3538 0.8708 0.0410 0.0425 0.0337 0.1519 0.7488 0.3693 suite8 14 0.3538 0.8706 0.0456 0.0567 0.0528 0.1517 0.7513 0.3697 suite10_final 14 0.3532 0.8705 0.0409 0.0432 0.0359 0.1520 0.7492 0.3695 v2 1 0.3518 0.8561 0.0990 0.1436 0.1436 0.2156 0.7076 0.4499 suite7 14 0.3518 0.8681 0.0447 0.0530 0.0484 0.1498 0.7409 0.3646 suite9_candidate 14 0.3505 0.8698 0.0359 0.0283 0.0153 0.1511 0.7483 0.3676 shared 14 0.3504 0.8689 0.0461 0.0588 0.0540 0.1514 0.7467 0.3686 suite10_candidate 14 0.3489 0.8700 0.0336 0.0237 0.0137 0.1522 0.7537 0.3703 suite9_final 14 0.3487 0.8678 0.0407 0.0411 0.0330 0.1499 0.7396 0.3641 suite13 14 0.3360 0.8463 0.0526 0.0553 0.0369 0.1456 0.7063 0.3519 suite12 14 0.3259 0.8481 0.0530 0.0565 0.0377 0.1474 0.7213 0.3582 suite20_gru 14 0.3034 0.8611 0.0752 0.0978 0.0913 0.1467 0.7427 0.3613 suite20_transformer 14 0.2941 0.8665 0.0840 0.1069 0.1041 0.1471 0.7420 0.3614 suite20_flat 14 0.2791 0.8592 0.0929 0.1358 0.1350 0.1458 0.7286 0.3560 SUITE21 HOLDOUT MACRO SUMMARY — F6/F7 -------------------------------------------------------------------------------- model faculties mean_fold_ap mean_fold_auc mean_fold_brier mean_fold_ece10 mean_fold_calibration_gap mean_fold_precision_at_20 mean_fold_recall_at_20 mean_fold_fbeta_at_20 council 1 0.4449 0.8330 0.0874 0.0154 0.0138 0.3813 0.6170 0.5442 v2 1 0.4227 0.8266 0.1428 0.1623 0.1623 0.3701 0.5983 0.5279 suite14 14 0.3602 0.8504 0.0687 0.0738 0.0697 0.2333 0.6866 0.4523 suite17 14 0.3578 0.8528 0.0532 0.0251 0.0027 0.2340 0.6874 0.4535 suite18 14 0.3578 0.8529 0.0533 0.0253 0.0025 0.2352 0.6893 0.4553 suite21_guarded_temporal_residual 14 0.3571 0.8469 0.0531 0.0238 0.0053 0.2308 0.6763 0.4464 suite10_candidate 14 0.3570 0.8452 0.0534 0.0263 0.0065 0.2320 0.6789 0.4484 suite16 14 0.3561 0.8520 0.0728 0.0889 0.0871 0.2341 0.6882 0.4539 suite10_final 14 0.3559 0.8448 0.0608 0.0427 0.0268 0.2313 0.6732 0.4457 suite9_candidate 14 0.3547 0.8457 0.0567 0.0339 0.0110 0.2317 0.6759 0.4472 suite21_flat_residual 14 0.3546 0.8479 0.0530 0.0242 0.0031 0.2327 0.6828 0.4504 suite21_temporal_residual 14 0.3541 0.8473 0.0531 0.0234 0.0041 0.2329 0.6831 0.4508 suite9_final 14 0.3541 0.8451 0.0605 0.0422 0.0293 0.2318 0.6740 0.4466 suite8 14 0.3535 0.8440 0.0688 0.0639 0.0578 0.2306 0.6759 0.4463 suite7 14 0.3533 0.8438 0.0690 0.0643 0.0588 0.2303 0.6754 0.4457 suite11_final 14 0.3532 0.8452 0.0609 0.0414 0.0214 0.2314 0.6752 0.4467 shared 14 0.3530 0.8467 0.0709 0.0721 0.0681 0.2309 0.6804 0.4485 suite11_candidate 14 0.3513 0.8447 0.0535 0.0249 0.0002 0.2316 0.6785 0.4480 committee 14 0.3502 0.8434 0.0533 0.0240 0.0016 0.2311 0.6756 0.4465 suite13 14 0.3430 0.8409 0.0816 0.0817 0.0660 0.2285 0.6691 0.4420 suite12 14 0.3377 0.8407 0.0829 0.0881 0.0708 0.2286 0.6767 0.4453 suite20_transformer 14 0.3249 0.8396 0.0999 0.1189 0.1144 0.2227 0.6666 0.4365 suite20_gru 14 0.3089 0.8360 0.0897 0.1059 0.1032 0.2200 0.6593 0.4318 suite20_flat 14 0.3078 0.8329 0.1119 0.1516 0.1514 0.2229 0.6576 0.4333 SUITE21 PRIMARY vs ALL AVAILABLE PREVIOUS VERSIONS — HOLDOUT -------------------------------------------------------------------------------- reference faculties candidate_macro_ap reference_macro_ap mean_delta_ap median_delta_ap q25_delta_ap worst_delta_ap ap_wins ap_win_rate mean_delta_f2 f2_wins committee 14 0.3571 0.3502 0.0069 0.0051 0.0011 -0.0034 12 0.8571 -0.0001 9 council 1 0.4192 0.4449 -0.0257 -0.0257 -0.0257 -0.0257 0 0.0000 -0.0122 0 shared 14 0.3571 0.3530 0.0040 0.0015 -0.0016 -0.0083 9 0.6429 -0.0021 4 suite10_candidate 14 0.3571 0.3570 0.0001 0.0014 -0.0068 -0.0153 8 0.5714 -0.0020 5 suite10_final 14 0.3571 0.3559 0.0011 0.0009 -0.0051 -0.0158 8 0.5714 0.0007 8 suite11_candidate 14 0.3571 0.3513 0.0057 0.0060 -0.0006 -0.0038 10 0.7143 -0.0016 6 suite11_final 14 0.3571 0.3532 0.0038 0.0016 -0.0036 -0.0106 8 0.5714 -0.0003 7 suite12 14 0.3571 0.3377 0.0193 0.0228 -0.0029 -0.0492 10 0.7143 0.0011 7 suite13 14 0.3571 0.3430 0.0141 0.0142 -0.0003 -0.0461 10 0.7143 0.0045 7 suite14 14 0.3571 0.3602 -0.0032 -0.0046 -0.0060 -0.0162 4 0.2857 -0.0059 3 suite16 14 0.3571 0.3561 0.0010 -0.0003 -0.0073 -0.0149 7 0.5000 -0.0074 1 suite17 14 0.3571 0.3578 -0.0007 -0.0025 -0.0057 -0.0121 6 0.4286 -0.0071 2 suite18 14 0.3571 0.3578 -0.0007 -0.0001 -0.0077 -0.0125 7 0.5000 -0.0089 2 suite20_flat 14 0.3571 0.3078 0.0493 0.0507 0.0252 -0.0212 13 0.9286 0.0132 8 suite20_gru 14 0.3571 0.3089 0.0482 0.0490 0.0341 0.0143 14 1.0000 0.0146 9 suite20_transformer 14 0.3571 0.3249 0.0321 0.0237 0.0141 -0.0182 12 0.8571 0.0099 8 suite21_flat_residual 14 0.3571 0.3546 0.0024 0.0019 -0.0011 -0.0065 10 0.7143 -0.0040 4 suite21_temporal_residual 14 0.3571 0.3541 0.0029 0.0025 -0.0046 -0.0126 9 0.6429 -0.0044 5 suite7 14 0.3571 0.3533 0.0037 -0.0002 -0.0061 -0.0139 7 0.5000 0.0007 7 suite8 14 0.3571 0.3535 0.0036 0.0018 -0.0050 -0.0140 8 0.5714 0.0001 5 suite9_candidate 14 0.3571 0.3547 0.0024 0.0023 -0.0041 -0.0214 8 0.5714 -0.0008 7 suite9_final 14 0.3571 0.3541 0.0030 -0.0001 -0.0038 -0.0136 6 0.4286 -0.0002 8 v2 1 0.4192 0.4227 -0.0035 -0.0035 -0.0035 -0.0035 0 0.0000 0.0042 1 SUITE21 OPERATIONAL HOLDOUT SUMMARY — THRESHOLDS FROZEN ON F1..F5 -------------------------------------------------------------------------------- model faculties mean_ap mean_fixed_precision mean_fixed_recall mean_fixed_fbeta mean_fixed_flag_rate mean_drift_precision mean_drift_recall mean_drift_fbeta mean_drift_flag_rate council 1 0.4449 0.3313 0.7164 0.5795 0.2682 0.3656 0.6611 0.5681 0.2312 v2 1 0.4227 0.3217 0.7013 0.5658 0.2699 0.3324 0.6714 0.5567 0.2581 suite14 14 0.3602 0.2282 0.6870 0.4610 0.2336 0.3048 0.5822 0.4608 0.1526 suite17 14 0.3578 0.2843 0.5969 0.4620 0.1653 0.3212 0.5363 0.4610 0.1231 suite18 14 0.3578 0.2851 0.5956 0.4648 0.1619 0.3194 0.5338 0.4600 0.1236 suite21_guarded_temporal_residual 14 0.3571 0.2745 0.6059 0.4673 0.1695 0.3023 0.5485 0.4591 0.1324 suite10_candidate 14 0.3570 0.2849 0.5732 0.4504 0.1552 0.3105 0.5433 0.4521 0.1322 suite16 14 0.3561 0.3437 0.4843 0.4362 0.1095 0.3093 0.5353 0.4594 0.1260 suite10_final 14 0.3559 0.3312 0.4650 0.3825 0.1135 0.3252 0.5075 0.4361 0.1230 suite9_candidate 14 0.3547 0.2988 0.5525 0.4458 0.1468 0.3316 0.4966 0.4382 0.1135 suite21_flat_residual 14 0.3546 0.2720 0.6088 0.4732 0.1672 0.3050 0.5338 0.4555 0.1280 suite21_temporal_residual 14 0.3541 0.2671 0.6167 0.4729 0.1737 0.3088 0.5409 0.4558 0.1310 suite9_final 14 0.3541 0.3704 0.4128 0.3509 0.0944 0.3552 0.4611 0.4221 0.0971 suite8 14 0.3535 0.3182 0.5235 0.4477 0.1223 0.3266 0.4909 0.4351 0.1088 suite7 14 0.3533 0.3123 0.5325 0.4415 0.1320 0.3526 0.4664 0.4263 0.0965 suite11_final 14 0.3532 0.3616 0.3988 0.3547 0.0827 0.3424 0.4740 0.4287 0.1022 shared 14 0.3530 0.3097 0.5537 0.4579 0.1306 0.3363 0.4979 0.4413 0.1090 suite11_candidate 14 0.3513 0.2395 0.6577 0.4600 0.2153 0.3037 0.5305 0.4486 0.1318 committee 14 0.3502 0.2619 0.5999 0.4623 0.1695 0.3036 0.5318 0.4524 0.1267 suite13 14 0.3430 0.2823 0.5869 0.4569 0.1659 0.3355 0.4828 0.4360 0.1026 suite12 14 0.3377 0.2651 0.6211 0.4324 0.2346 0.3012 0.5162 0.4373 0.1247 suite20_transformer 14 0.3249 0.2973 0.5007 0.4181 0.1291 0.2570 0.5915 0.4518 0.1662 suite20_gru 14 0.3089 0.3285 0.4069 0.3673 0.1068 0.2805 0.5240 0.4351 0.1386 suite20_flat 14 0.3078 0.3178 0.4357 0.3880 0.1098 0.2703 0.5421 0.4369 0.1483 SUITE21 AUTHORITY SELECTION -------------------------------------------------------------------------------- fold variant selected_epochs selected_alpha validation_robust_score 1 flat_residual 1 0.2500 0.0009 1 temporal_residual 3 0.2500 0.0015 1 guarded_temporal_residual 1 0.5000 0.0037 2 flat_residual 1 1.0000 0.0153 2 temporal_residual 1 0.5000 0.0083 2 guarded_temporal_residual 2 0.7500 0.0111 3 flat_residual 4 0.5000 0.0074 3 temporal_residual 8 0.5000 0.0107 3 guarded_temporal_residual 1 0.5000 0.0061 4 flat_residual 3 0.2500 0.0090 4 temporal_residual 1 0.7500 0.0105 4 guarded_temporal_residual 3 0.5000 0.0122 5 flat_residual 1 0.7500 0.0084 5 temporal_residual 2 0.7500 0.0085 5 guarded_temporal_residual 1 1.0000 0.0108 6 flat_residual 1 0.7500 0.0088 6 temporal_residual 2 0.5000 0.0095 6 guarded_temporal_residual 5 0.5000 0.0145 7 flat_residual 1 1.0000 0.0070 7 temporal_residual 1 1.0000 0.0093 7 guarded_temporal_residual 1 1.0000 0.0085 HISTORICAL COUNCIL — FOLD-BY-FOLD AP/AUC ON 13-FACULTY OVERLAP -------------------------------------------------------------------------------- fold candidate_ap candidate_auc council_ap council_auc delta_ap delta_auc 1 0.2451 0.8308 0.2624 0.8355 -0.0173 -0.0047 2 0.4946 0.9247 0.4952 0.9195 -0.0006 0.0052 3 0.3057 0.8764 0.2936 0.8707 0.0121 0.0056 4 0.4349 0.9189 0.4303 0.9132 0.0046 0.0057 5 0.2979 0.8713 0.2973 0.8689 0.0006 0.0023 6 0.3734 0.8612 0.3733 0.8620 0.0001 -0.0008 7 0.3560 0.8343 0.3551 0.8376 0.0008 -0.0033 HISTORICAL COUNCIL REFERENCE — 13 FACULTIES ONLY (ENGLISH MISSING) -------------------------------------------------------------------------------- source faculties macro_ap macro_auc macro_fixed_f2 macro_drift_f2 suite21_primary 13 0.3647 0.8478 0.4786 0.4690 council_historical_13_faculty 13 0.3642 0.8498 0.4436 0.4502 SUITE21 PRIMARY COMMITTEE RETENTION — HOLDOUT -------------------------------------------------------------------------------- fold faculty committee_ap candidate_ap delta_ap probability_mae rank_corr top20_overlap failure_mean_delta normal_mean_delta 6 khoa_cong_nghe_thong_tin 0.3724 0.3805 0.0081 0.0024 0.9986 0.9761 0.0039 -0.0001 6 khoa_quan_tri_kinh_doanh_va_du_lich 0.3777 0.3822 0.0046 0.0015 0.9992 0.9785 0.0004 -0.0009 6 khoa_quoc_te_hoc 0.2778 0.2754 -0.0025 0.0022 0.9988 0.9777 -0.0037 -0.0013 6 khoa_tieng_anh 0.2882 0.2903 0.0021 0.0026 0.9981 0.9605 0.0013 -0.0003 6 khoa_tieng_bo_dao_nha 0.5210 0.5200 -0.0010 0.0059 0.9966 0.9494 0.0057 0.0023 6 khoa_tieng_duc 0.4558 0.4626 0.0067 0.0030 0.9931 0.9348 0.0048 -0.0003 6 khoa_tieng_han_quoc 0.2531 0.2630 0.0099 0.0025 0.9978 0.9588 0.0025 -0.0006 6 khoa_tieng_italia 0.2286 0.2308 0.0022 0.0044 0.9821 0.9654 0.0037 0.0014 6 khoa_tieng_nga 0.4801 0.4812 0.0011 0.0018 0.9977 0.9755 0.0008 0.0000 6 khoa_tieng_nhat 0.1763 0.1847 0.0085 0.0032 0.9932 0.9269 0.0081 0.0004 6 khoa_tieng_phap 0.2791 0.2803 0.0012 0.0015 0.9948 0.9294 -0.0000 0.0001 6 khoa_tieng_tay_ban_nha 0.3718 0.3824 0.0106 0.0041 0.9936 0.9775 0.0068 -0.0006 6 khoa_tieng_trung_quoc 0.3394 0.3587 0.0194 0.0029 0.9942 0.9489 0.0166 0.0013 6 khoa_viet_nam_hoc 0.6504 0.6520 0.0016 0.0076 0.9953 0.9752 0.0060 0.0070 7 khoa_cong_nghe_thong_tin 0.4527 0.4578 0.0051 0.0212 0.9677 0.8412 0.0334 0.0161 7 khoa_quan_tri_kinh_doanh_va_du_lich 0.4207 0.4093 -0.0114 0.0167 0.9742 0.8454 0.0242 0.0103 7 khoa_quoc_te_hoc 0.2446 0.2477 0.0032 0.0061 0.9779 0.8113 0.0125 0.0032 7 khoa_tieng_anh 0.2209 0.2259 0.0050 0.0053 0.9938 0.9295 -0.0028 -0.0046 7 khoa_tieng_bo_dao_nha 0.4075 0.4220 0.0145 0.0159 0.9739 0.9380 0.0205 0.0016 7 khoa_tieng_duc 0.2956 0.3094 0.0138 0.0089 0.9810 0.9125 0.0132 -0.0001 7 khoa_tieng_han_quoc 0.3786 0.4093 0.0307 0.0067 0.9901 0.8879 0.0242 -0.0003 7 khoa_tieng_italia 0.3767 0.3805 0.0038 0.0065 0.9828 0.9348 0.0019 -0.0027 7 khoa_tieng_nga 0.2739 0.2745 0.0006 0.0156 0.9620 0.8779 0.0196 0.0038 7 khoa_tieng_nhat 0.2195 0.2386 0.0191 0.0118 0.9726 0.9088 0.0256 0.0020 7 khoa_tieng_phap 0.2748 0.2717 -0.0031 0.0038 0.9875 0.8696 -0.0007 -0.0016 7 khoa_tieng_tay_ban_nha 0.4033 0.4355 0.0323 0.0172 0.9778 0.8849 0.0431 0.0046 7 khoa_tieng_trung_quoc 0.1597 0.1648 0.0051 0.0092 0.9824 0.9031 0.0175 -0.0003 7 khoa_viet_nam_hoc 0.6040 0.6065 0.0025 0.0456 0.9788 0.9337 0.0551 0.0429 SUITE21 PRIMARY RESIDUAL / GUARD DIAGNOSTICS — HOLDOUT -------------------------------------------------------------------------------- fold faculty authority_mean authority_failure_mean authority_normal_mean guard_mean correction_abs_mean failure_correction_mean failure_negative_correction_fraction rescued_failures lost_failures removed_false_positives 6 khoa_cong_nghe_thong_tin 0.8725 0.5811 0.9047 0.8886 0.0489 0.0286 0.3481 6 5 22 6 khoa_quan_tri_kinh_doanh_va_du_lich 0.8730 0.4774 0.9073 0.9120 0.0440 0.0013 0.5363 3 1 31 6 khoa_quoc_te_hoc 0.9630 0.7242 0.9689 0.9520 0.0738 -0.0143 0.5500 0 0 11 6 khoa_tieng_anh 0.9743 0.5875 0.9820 0.9565 0.0730 0.0210 0.2869 1 0 49 6 khoa_tieng_bo_dao_nha 0.8364 0.3669 0.8782 0.8771 0.1031 0.0375 0.2877 1 0 9 6 khoa_tieng_duc 0.9031 0.3519 0.9295 0.9278 0.1058 0.0636 0.2500 1 0 24 6 khoa_tieng_han_quoc 0.9672 0.8027 0.9736 0.9611 0.0767 0.0325 0.4634 0 2 25 6 khoa_tieng_italia 0.8508 0.3916 0.8732 0.8987 0.1352 0.0477 0.1915 1 1 13 6 khoa_tieng_nga 0.8074 0.4335 0.8542 0.8812 0.0533 0.0114 0.3512 1 1 8 6 khoa_tieng_nhat 0.9428 0.7005 0.9515 0.9504 0.1235 0.1081 0.4091 1 0 37 6 khoa_tieng_phap 0.9480 0.4080 0.9602 0.9529 0.0808 0.0186 0.3200 1 0 32 6 khoa_tieng_tay_ban_nha 0.8450 0.4118 0.8745 0.8919 0.0934 0.0742 0.3176 0 0 6 6 khoa_tieng_trung_quoc 0.9523 0.6884 0.9614 0.9438 0.0909 0.1626 0.1737 0 0 51 6 khoa_viet_nam_hoc 0.6964 0.1540 0.7889 0.7923 0.1149 0.0649 0.1250 0 2 1 7 khoa_cong_nghe_thong_tin 0.7827 0.6340 0.8093 0.8663 0.2429 0.2221 0.1527 56 50 170 7 khoa_quan_tri_kinh_doanh_va_du_lich 0.8104 0.6201 0.8376 0.8701 0.2137 0.1548 0.2538 40 67 211 7 khoa_quoc_te_hoc 0.9240 0.7687 0.9315 0.9625 0.2093 0.1364 0.3203 6 9 95 7 khoa_tieng_anh 0.9252 0.6956 0.9323 0.9460 0.1940 -0.1037 0.7760 5 5 113 7 khoa_tieng_bo_dao_nha 0.7731 0.5820 0.7988 0.8238 0.1936 0.0715 0.3497 1 1 14 7 khoa_tieng_duc 0.8912 0.5929 0.9064 0.9249 0.2278 0.0679 0.3423 3 1 39 7 khoa_tieng_han_quoc 0.9300 0.6250 0.9415 0.9334 0.1716 0.1483 0.2000 6 2 75 7 khoa_tieng_italia 0.8369 0.6424 0.8594 0.8924 0.1369 -0.0102 0.6024 5 4 28 7 khoa_tieng_nga 0.7465 0.5345 0.7650 0.8322 0.1862 0.1061 0.3312 5 5 42 7 khoa_tieng_nhat 0.9049 0.7250 0.9147 0.9166 0.2578 0.1394 0.2276 0 1 50 7 khoa_tieng_phap 0.9255 0.6945 0.9353 0.9535 0.1731 -0.0918 0.6387 2 7 69 7 khoa_tieng_tay_ban_nha 0.8008 0.5863 0.8239 0.8648 0.2262 0.2245 0.2316 7 6 39 7 khoa_tieng_trung_quoc 0.9137 0.7029 0.9208 0.9327 0.2067 0.0785 0.4332 7 4 125 7 khoa_viet_nam_hoc 0.7472 0.4879 0.8025 0.7123 0.3771 0.2931 0.0523 5 4 9 SUITE21 PRIMARY HISTORY PERTURBATION — HOLDOUT -------------------------------------------------------------------------------- fold faculty perturbation base_ap perturbed_ap delta_ap mean_abs_probability_change 6 khoa_cong_nghe_thong_tin reverse_history 0.3805 0.3755 -0.0050 0.0032 6 khoa_quan_tri_kinh_doanh_va_du_lich reverse_history 0.3822 0.3828 0.0006 0.0024 6 khoa_quoc_te_hoc reverse_history 0.2754 0.2721 -0.0033 0.0019 6 khoa_tieng_anh reverse_history 0.2903 0.2918 0.0015 0.0031 6 khoa_tieng_bo_dao_nha reverse_history 0.5200 0.5218 0.0018 0.0074 6 khoa_tieng_duc reverse_history 0.4626 0.4629 0.0003 0.0039 6 khoa_tieng_han_quoc reverse_history 0.2630 0.2672 0.0041 0.0021 6 khoa_tieng_italia reverse_history 0.2308 0.2302 -0.0006 0.0054 6 khoa_tieng_nga reverse_history 0.4812 0.4802 -0.0011 0.0045 6 khoa_tieng_nhat reverse_history 0.1847 0.1804 -0.0043 0.0043 6 khoa_tieng_phap reverse_history 0.2803 0.2821 0.0018 0.0022 6 khoa_tieng_tay_ban_nha reverse_history 0.3824 0.3772 -0.0052 0.0063 6 khoa_tieng_trung_quoc reverse_history 0.3587 0.3504 -0.0084 0.0047 6 khoa_viet_nam_hoc reverse_history 0.6520 0.6500 -0.0020 0.0089 6 khoa_cong_nghe_thong_tin zero_history 0.3805 0.3724 -0.0081 0.0024 6 khoa_quan_tri_kinh_doanh_va_du_lich zero_history 0.3822 0.3777 -0.0046 0.0015 6 khoa_quoc_te_hoc zero_history 0.2754 0.2778 0.0025 0.0022 6 khoa_tieng_anh zero_history 0.2903 0.2882 -0.0021 0.0026 6 khoa_tieng_bo_dao_nha zero_history 0.5200 0.5210 0.0010 0.0059 6 khoa_tieng_duc zero_history 0.4626 0.4558 -0.0067 0.0030 6 khoa_tieng_han_quoc zero_history 0.2630 0.2531 -0.0099 0.0025 6 khoa_tieng_italia zero_history 0.2308 0.2286 -0.0022 0.0044 6 khoa_tieng_nga zero_history 0.4812 0.4801 -0.0011 0.0018 6 khoa_tieng_nhat zero_history 0.1847 0.1763 -0.0085 0.0032 6 khoa_tieng_phap zero_history 0.2803 0.2791 -0.0012 0.0015 6 khoa_tieng_tay_ban_nha zero_history 0.3824 0.3718 -0.0106 0.0041 6 khoa_tieng_trung_quoc zero_history 0.3587 0.3394 -0.0194 0.0029 6 khoa_viet_nam_hoc zero_history 0.6520 0.6504 -0.0016 0.0076 7 khoa_cong_nghe_thong_tin reverse_history 0.4578 0.4531 -0.0047 0.0204 7 khoa_quan_tri_kinh_doanh_va_du_lich reverse_history 0.4093 0.4160 0.0067 0.0168 7 khoa_quoc_te_hoc reverse_history 0.2477 0.2370 -0.0108 0.0054 7 khoa_tieng_anh reverse_history 0.2259 0.2285 0.0026 0.0014 7 khoa_tieng_bo_dao_nha reverse_history 0.4220 0.4182 -0.0037 0.0064 7 khoa_tieng_duc reverse_history 0.3094 0.3032 -0.0061 0.0033 7 khoa_tieng_han_quoc reverse_history 0.4093 0.4088 -0.0005 0.0028 7 khoa_tieng_italia reverse_history 0.3805 0.3810 0.0005 0.0032 7 khoa_tieng_nga reverse_history 0.2745 0.2727 -0.0018 0.0068 7 khoa_tieng_nhat reverse_history 0.2386 0.2267 -0.0120 0.0067 7 khoa_tieng_phap reverse_history 0.2717 0.2761 0.0044 0.0016 7 khoa_tieng_tay_ban_nha reverse_history 0.4355 0.4296 -0.0059 0.0068 7 khoa_tieng_trung_quoc reverse_history 0.1648 0.1713 0.0066 0.0030 7 khoa_viet_nam_hoc reverse_history 0.6065 0.5968 -0.0097 0.0250 7 khoa_cong_nghe_thong_tin zero_history 0.4578 0.4527 -0.0051 0.0212 7 khoa_quan_tri_kinh_doanh_va_du_lich zero_history 0.4093 0.4207 0.0114 0.0167 7 khoa_quoc_te_hoc zero_history 0.2477 0.2446 -0.0032 0.0061 7 khoa_tieng_anh zero_history 0.2259 0.2209 -0.0050 0.0053 7 khoa_tieng_bo_dao_nha zero_history 0.4220 0.4075 -0.0145 0.0159 7 khoa_tieng_duc zero_history 0.3094 0.2956 -0.0138 0.0089 7 khoa_tieng_han_quoc zero_history 0.4093 0.3786 -0.0307 0.0067 7 khoa_tieng_italia zero_history 0.3805 0.3767 -0.0038 0.0065 7 khoa_tieng_nga zero_history 0.2745 0.2739 -0.0006 0.0156 7 khoa_tieng_nhat zero_history 0.2386 0.2195 -0.0191 0.0118 7 khoa_tieng_phap zero_history 0.2717 0.2748 0.0031 0.0038 7 khoa_tieng_tay_ban_nha zero_history 0.4355 0.4033 -0.0323 0.0172 7 khoa_tieng_trung_quoc zero_history 0.1648 0.1597 -0.0051 0.0092 7 khoa_viet_nam_hoc zero_history 0.6065 0.6040 -0.0025 0.0456 [HISTORY PERTURBATION SUMMARY] delta_ap_mean delta_ap_median harmed_cells cells mean_abs_change perturbation reverse_history -0.0019 -0.0014 17 28 0.0061 zero_history -0.0069 -0.0048 24 28 0.0084 ============================================================================================================== OPERATIONAL CONFUSION MATRICES — khoa_cong_nghe_thong_tin — POOLED F6/F7 ============================================================================================================== suite21_guarded_temporal_residual — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,332 2,634 Failure <5 448 1,163 ---------------------------------------------------------------------------- Precision=0.3063 Recall=0.7219 F2=0.5678 Flag=30.19% suite21_guarded_temporal_residual — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 9,024 1,942 Failure <5 575 1,036 ---------------------------------------------------------------------------- Precision=0.3479 Recall=0.6431 F2=0.5498 Flag=23.68% committee — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,511 2,455 Failure <5 459 1,152 ---------------------------------------------------------------------------- Precision=0.3194 Recall=0.7151 F2=0.5731 Flag=28.68% committee — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,934 2,032 Failure <5 538 1,073 ---------------------------------------------------------------------------- Precision=0.3456 Recall=0.6660 F2=0.5618 Flag=24.69% suite18 — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,335 2,631 Failure <5 438 1,173 ---------------------------------------------------------------------------- Precision=0.3084 Recall=0.7281 F2=0.5723 Flag=30.25% suite18 — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,745 2,221 Failure <5 512 1,099 ---------------------------------------------------------------------------- Precision=0.3310 Recall=0.6822 F2=0.5628 Flag=26.40% suite20_transformer — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 9,820 1,146 Failure <5 828 783 ---------------------------------------------------------------------------- Precision=0.4059 Recall=0.4860 F2=0.4676 Flag=15.34% suite20_transformer — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,498 2,468 Failure <5 513 1,098 ---------------------------------------------------------------------------- Precision=0.3079 Recall=0.6816 F2=0.5485 Flag=28.35% v2 — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,660 2,306 Failure <5 489 1,122 ---------------------------------------------------------------------------- Precision=0.3273 Recall=0.6965 F2=0.5683 Flag=27.26% v2 — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,735 2,231 Failure <5 501 1,110 ---------------------------------------------------------------------------- Precision=0.3322 Recall=0.6890 F2=0.5672 Flag=26.56% council — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 8,712 2,254 Failure <5 460 1,151 ---------------------------------------------------------------------------- Precision=0.3380 Recall=0.7145 F2=0.5843 Flag=27.07% council — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 9,064 1,902 Failure <5 520 1,091 ---------------------------------------------------------------------------- Precision=0.3645 Recall=0.6772 F2=0.5780 Flag=23.80% ============================================================================================================== OVERALL 14-FACULTY POOLED CONFUSIONS — F6/F7 ============================================================================================================== suite21_guarded_temporal_residual — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 79,961 10,469 Failure <5 2,544 4,030 ---------------------------------------------------------------------------- Precision=0.2780 Recall=0.6130 F2=0.4939 Flag=14.95% suite21_guarded_temporal_residual — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 82,461 7,969 Failure <5 2,952 3,622 ---------------------------------------------------------------------------- Precision=0.3125 Recall=0.5510 F2=0.4780 Flag=11.95% committee — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 79,962 10,468 Failure <5 2,508 4,066 ---------------------------------------------------------------------------- Precision=0.2798 Recall=0.6185 F2=0.4979 Flag=14.98% committee — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 82,442 7,988 Failure <5 2,921 3,653 ---------------------------------------------------------------------------- Precision=0.3138 Recall=0.5557 F2=0.4815 Flag=12.00% suite18 — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 81,310 9,120 Failure <5 2,757 3,817 ---------------------------------------------------------------------------- Precision=0.2950 Recall=0.5806 F2=0.4865 Flag=13.34% suite18 — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 82,936 7,494 Failure <5 2,963 3,611 ---------------------------------------------------------------------------- Precision=0.3252 Recall=0.5493 F2=0.4827 Flag=11.45% suite20_transformer — fixed ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 83,722 6,708 Failure <5 3,422 3,152 ---------------------------------------------------------------------------- Precision=0.3197 Recall=0.4795 F2=0.4359 Flag=10.16% suite20_transformer — drift ---------------------------------------------------------------------------- Actual / Predicted Normal >=5 Failure <5 ---------------------------------------------------------------------------- Normal >=5 79,036 11,394 Failure <5 2,489 4,085 ---------------------------------------------------------------------------- Precision=0.2639 Recall=0.6214 F2=0.4889 Flag=15.96% REFERENCE COVERAGE -------------------------------------------------------------------------------- model faculties folds faculty_fold_cells committee 14 7 98 shared 14 7 98 suite10_candidate 14 7 98 suite10_final 14 7 98 suite11_candidate 14 7 98 suite11_final 14 7 98 suite12 14 7 98 suite13 14 7 98 suite14 14 7 98 suite16 14 7 98 suite17 14 7 98 suite18 14 7 98 suite20_flat 14 7 98 suite20_gru 14 7 98 suite20_transformer 14 7 98 suite21_flat_residual 14 7 98 suite21_guarded_temporal_residual 14 7 98 suite21_temporal_residual 14 7 98 suite7 14 7 98 suite8 14 7 98 suite9_candidate 14 7 98 suite9_final 14 7 98 council 1 7 7 v2 1 7 7 [DONE] Suite21 complete in 1.19 hours. Results: /root/disk_3tb/earlywarning/30072026/results/suite21_guarded_temporal_residual