[START] 2026-10-08T10:22:31+00:00 Suite28 DRRM [RUN] here=/root/disk_3tb/earlywarning/30072026/suite28 root=/root/disk_3tb/earlywarning/30072026 python=/root/miniconda3/envs/Huy/bin/python3 [RUN] arguments=--faculties khoa_cong_nghe_thong_tin,khoa_tieng_nhat --folds 5 --dev-folds 4,5 --max-epochs 1 --batch-students 32 --bootstrap 20 --skip-flat-control --out /root/disk_3tb/earlywarning/30072026/results/suite28_PILOT_NOT_COMPARABLE [SUITE28 PREFLIGHT] python= 3.13.5 OS= Linux-5.15.0-191-generic-x86_64-with-glibc2.35 [PACKAGE] numpy 2.2.6 [PACKAGE] scipy 1.16.0 [PACKAGE] pandas 2.3.1 [PACKAGE] sklearn 1.7.1 [PACKAGE] joblib 1.5.1 [PACKAGE] torch 2.7.1+cu118 [ACCELERATOR] cuda= True device= Tesla V100-SXM2-32GB [FOLD 1] canonical=14 structured_missing=0 [FOLD 2] canonical=14 structured_missing=0 [FOLD 3] canonical=14 structured_missing=0 [FOLD 4] canonical=14 structured_missing=0 [FOLD 5] canonical=14 structured_missing=0 [FOLD 6] canonical=14 structured_missing=0 [FOLD 7] canonical=14 structured_missing=0 [SUITE28 PREFLIGHT] BASIC PATH/DEPENDENCY CHECK PASSED [NOTE] For schema, row alignment, grade and student-context checks use train_suite28_drrm.py --preflight-only [PRECHECK] auditing student-semester context invariance across all historical folds... [CONTEXT AUDIT] F1 candidate=24 invariant=24 groups=6299 unstable={} [CONTEXT AUDIT] F2 candidate=24 invariant=24 groups=7706 unstable={} [CONTEXT AUDIT] F3 candidate=24 invariant=24 groups=9007 unstable={} [CONTEXT AUDIT] F4 candidate=24 invariant=24 groups=10621 unstable={} [CONTEXT AUDIT] F5 candidate=24 invariant=24 groups=12096 unstable={} [CONTEXT AUDIT] F6 candidate=24 invariant=24 groups=13720 unstable={} [CONTEXT AUDIT] F7 candidate=24 invariant=24 groups=15351 unstable={} [CONTEXT OWNERSHIP] frozen 24 fields: ['prev_semester_NHHK', 'prev_semester_ordinal', 'prev_passed_credits', 'prev_failed_credits', 'prev_outstanding_failed_credits', 'prev_semester_gpa10', 'prev_cumulative_gpa10', 'hist_semester_count', 'hist_semester_gpa10_mean', 'hist_cumulative_gpa10_mean', 'hist_registered_credits_mean', 'recent3_semester_count', 'recent3_semester_gpa10_mean', 'hist_warning_count', 'hist_warning_rate', 'hist_warning_consecutive_max', 'hist_warning_in_last_semester', 'hist_warning_in_last2_consecutive', 'recent3_warning_count', 'hist_warning_credit_debt_count', 'hist_warning_gpa_sem_count', 'hist_warning_gpa_cum_count', 'target_semester_registered_credits', 'target_semester_difficulty_weighted_workload'] [FEATURE PROVENANCE WARNING] original 136 common upstream features cannot be fully provenance-certified from these caches. Suite28 rebuilds course outcome aggregates as-of and never passes query grade or failure to inference. [ENV] Python=3.13.5 torch=2.7.1+cu118 device=cuda faculties=2 folds=[5] [SCIENTIFIC CONTRACT] B primary=DRRM; A=compensatory MIRT; contextual-only is RETRAINED; all learn without Committee input. [HOLDOUT STATUS] Historical F6/F7 have informed prior research and are not fresh prospective confirmation. [GRADES] Consistent ordinal [<5,5-6,6-7,7-8,8-9,9-10]; failure equals first bin. No same-semester outcomes before prediction. ================================================================================================================================== [DEVELOPMENT ORIGIN] F4 — independent earlier-only fitting ================================================================================================================================== [EVENT IDENTITY AUDIT] F4 coarse student-course-semester duplicate rows=0 / 62106 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F4 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F4 train_rows=62,106 target_rows=7,485 contexts=24 common_raw_full=208 FOLD 4 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 4 khoa_cong_nghe_thong_tin 5072 244 4828 0.048107 0.997831 0.958399 True 4 khoa_tieng_nhat 2413 38 2375 0.015748 0.998342 0.928305 True [SELECT] fold=4 model=suite28_context_only epoch=1/1 loss=1.75411 AP=0.10528 F2@20=0.21760 steps=88 sec=10.7 gradients={'context': 0.02868317998945713, 'course_meta': 0.015792492777109146, 'course_id': 0.012784934602677822, 'retention': 0.00760245556011796, 'grade_threshold': 0.16044726967811584} [SELECT] fold=4 model=suite28_mirt epoch=1/1 loss=1.74916 AP=0.17816 F2@20=0.26568 steps=88 sec=12.4 gradients={'context': 0.02800437994301319, 'course_meta': 0.014710234478116035, 'course_id': 0.024879295378923416, 'retention': 0.0053053661249578, 'grade_threshold': 0.16567450761795044} [SELECT] fold=4 model=suite28_drrm epoch=1/1 loss=1.73632 AP=0.13072 F2@20=0.25186 steps=88 sec=12.8 gradients={'context': 0.02561250887811184, 'course_meta': 0.015221673995256424, 'course_id': 0.012810775078833103, 'retention': 0.008387408219277859, 'grade_threshold': 0.16152988374233246} ================================================================================================================================== [DEVELOPMENT ORIGIN] F5 — independent earlier-only fitting ================================================================================================================================== [EVENT IDENTITY AUDIT] F5 coarse student-course-semester duplicate rows=0 / 69593 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F5 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F5 train_rows=69,593 target_rows=8,991 contexts=24 common_raw_full=208 FOLD 5 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 5 khoa_cong_nghe_thong_tin 6398 351 6047 0.054861 0.702094 0.940919 True 5 khoa_tieng_nhat 2593 69 2524 0.026610 0.711145 0.810644 True [SELECT] fold=5 model=suite28_context_only epoch=1/1 loss=1.76788 AP=0.20576 F2@20=0.32442 steps=88 sec=9.1 gradients={'context': 0.04072141647338867, 'course_meta': 0.009380906820297241, 'course_id': 0.011049780994653702, 'retention': 0.008582767099142075, 'grade_threshold': 0.11727815866470337} [SELECT] fold=5 model=suite28_mirt epoch=1/1 loss=1.75927 AP=0.22146 F2@20=0.34433 steps=88 sec=13.2 gradients={'context': 0.06251002103090286, 'course_meta': 0.009798594750463963, 'course_id': 0.02277901954948902, 'retention': 0.0057984101586043835, 'grade_threshold': 0.12683525681495667} [SELECT] fold=5 model=suite28_drrm epoch=1/1 loss=1.74815 AP=0.22464 F2@20=0.34224 steps=88 sec=13.5 gradients={'context': 0.037462446838617325, 'course_meta': 0.008661601692438126, 'course_id': 0.011245090514421463, 'retention': 0.008443119004368782, 'grade_threshold': 0.11879249662160873} [DEVELOPMENT FROZEN CHOICE] suite28_context_only -------------------------------------------------------------------------------- epoch AP F2 origins 1 0.15552 0.271013 2 [DEVELOPMENT FROZEN CHOICE] suite28_mirt -------------------------------------------------------------------------------- epoch AP F2 origins 1 0.19981 0.305006 2 [DEVELOPMENT FROZEN CHOICE] suite28_drrm -------------------------------------------------------------------------------- epoch AP F2 origins 1 0.177681 0.29705 2 [FROZEN DEV EPOCHS] {'suite28_context_only': 1, 'suite28_mirt': 1, 'suite28_drrm': 1} ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F5 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F5 coarse student-course-semester duplicate rows=0 / 69593 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F5 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F5 train_rows=69,593 target_rows=8,991 contexts=24 common_raw_full=208 FOLD 5 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 5 khoa_cong_nghe_thong_tin 6398 351 6047 0.054861 0.702094 0.940919 True 5 khoa_tieng_nhat 2593 69 2524 0.026610 0.711145 0.810644 True [PRODUCTION VARIANT] suite28_context_only epochs=1 [TRAIN] F5 suite28_context_only epoch=1/1 loss=1.76753 steps=88 sec=7.2 gradients={'context': 0.06851667165756226, 'course_meta': 0.017880260944366455, 'course_id': 0.011462855152785778, 'retention': 0.0012429235503077507, 'grade_threshold': 0.1211392879486084} [PREDICTION] F5 suite28_context_only macro AP=0.18479 AUC=0.77879 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F5 suite28_mirt epoch=1/1 loss=1.75874 steps=88 sec=10.0 gradients={'context': 0.059394046664237976, 'course_meta': 0.013635371811687946, 'course_id': 0.02281304821372032, 'retention': 0.0015518688596785069, 'grade_threshold': 0.1267094910144806} [PREDICTION] F5 suite28_mirt macro AP=0.20975 AUC=0.80165 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F5 suite28_drrm epoch=1/1 loss=1.74712 steps=88 sec=8.9 gradients={'context': 0.059431903064250946, 'course_meta': 0.014575467444956303, 'course_id': 0.011633677408099174, 'retention': 0.003802590537816286, 'grade_threshold': 0.12044604867696762} [GRADE LIKELIHOOD] observed_grade_rows= 8991 NLL= 1.7494374513626099 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F5 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 6398 351 12.534855 0.147148 2.506447 1.389542 0.566344 khoa_tieng_nhat 2593 69 9.636329 0.126598 2.099816 0.790298 0.571512 [PREDICTION] F5 suite28_drrm macro AP=0.20983 AUC=0.79990 F5 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 5 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.940600 552.690186 -2.651384 0.121091 0.046002 1.035211 0.000265 3.889576e-14 0.0 0.0 5 khoa_tieng_nhat bottleneck 2593 course_requirement 0.807944 336.185883 -2.707885 0.180623 0.059325 1.022529 0.000813 3.657069e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 5, 'rho_base': 0.822324812412262, 'process_noise_diag': [0.018632275983691216, 0.01883537694811821, 0.01888052374124527, 0.01883767358958721], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.8376778960227966, 0.8390018343925476, 0.8407800197601318, 0.8444209694862366], [0.8223374485969543, 0.832699179649353, 0.8402355909347534, 0.8468008637428284]]}] [INTERVENTION] F5 no_history mean_delta_ap=-0.03056 [INTERVENTION] F5 no_grade mean_delta_ap=-0.00848 [INTERVENTION] F5 mean_only mean_delta_ap=+0.00172 [INTERVENTION] F5 no_course_id mean_delta_ap=-0.00053 [INTERVENTION] F5 shuffle_historical_outcomes mean_delta_ap=-0.03238 F1–F7 MACRO PERFORMANCE -------------------------------------------------------------------------------- model ap auc brier ece10 fbeta_at_20 precision_at_20 recall_at_20 faculties cells council 0.291789 0.834749 0.052825 0.074601 0.454545 0.190625 0.695157 1 1 suite23 0.267828 0.839907 0.034025 0.006092 0.379267 0.140930 0.704323 2 2 v2 0.266546 0.829637 0.098030 0.124563 0.430328 0.180469 0.658120 1 1 suite24 0.266430 0.842405 0.034670 0.021547 0.393008 0.145982 0.730212 2 2 suite25 0.258001 0.838973 0.034062 0.006807 0.387419 0.143638 0.721665 2 2 suite22 0.257772 0.844276 0.034150 0.006329 0.382762 0.141685 0.714542 2 2 suite14 0.256614 0.832773 0.043337 0.037766 0.381480 0.141503 0.710145 2 2 suite21 0.256297 0.839861 0.034975 0.015793 0.388351 0.144028 0.723089 2 2 suite18 0.253098 0.840839 0.034711 0.013570 0.385556 0.142857 0.718816 2 2 committee 0.250742 0.832684 0.034467 0.010559 0.384625 0.142466 0.717391 2 2 suite17 0.249063 0.840969 0.034791 0.013608 0.387770 0.143429 0.724638 2 2 suite28_drrm 0.209830 0.799897 0.049283 0.113830 0.332575 0.122415 0.625604 2 2 suite28_mirt 0.209753 0.801649 0.051263 0.122128 0.339207 0.126973 0.623560 2 2 suite26 0.189306 0.807916 0.035575 0.011989 0.342463 0.129760 0.618791 2 2 suite28_context_only 0.184786 0.778792 0.049404 0.111210 0.316040 0.116191 0.595442 2 2 FROZEN HISTORICAL COUNCIL OVERLAP (NOT ROW-ALIGNED) -------------------------------------------------------------------------------- Empty DataFrame Columns: [fold, faculty, ap, fbeta_at_20, council_historical_ap, council_historical_fixed_f2, delta_ap] Index: [] [COUNCIL CAUTION] F6/F7 archive has 13 faculty cells, English absent; archival values may be rounded; no cluster CI is possible from this CSV alone. POSTFIT MECHANISM INTERVENTIONS BY HOLDOUT -------------------------------------------------------------------------------- Empty DataFrame Columns: [intervention, mean_delta_ap, harmed_cells, cells, score_mae] Index: [] HOLDOUT SUPPORT / COLD-START STRATA (EXCERPT) -------------------------------------------------------------------------------- Empty DataFrame Columns: [fold, faculty, stratum, group, model, rows, failures, ap, auc] Index: [] FROZEN HYPOTHESIS AUDIT -------------------------------------------------------------------------------- primary contextual fulltab_hgb mirt committee suite22_blend suite25_blend contextual_increment bottleneck_increment beats_2026_reference_by_0p02 reference_coverage_note 0.20983 0.184786 None 0.209753 0.250742 0.257772 0.258001 0.025044 0.000077 False Compare aligned cells, not unmatched aggregates. Historical F6/F7 previously used for design. [DONE] Suite28 finished elapsed_hours=0.052 results=/root/disk_3tb/earlywarning/30072026/results/suite28_PILOT_NOT_COMPARABLE [COMPLETE] 2026-10-08T10:25:56+00:00 Suite28 DRRM [START] 2026-10-08T10:31:45+00:00 Suite28 DRRM [RUN] here=/root/disk_3tb/earlywarning/30072026/suite28 root=/root/disk_3tb/earlywarning/30072026 python=/root/miniconda3/envs/Huy/bin/python3 [RUN] arguments= [SUITE28 PREFLIGHT] python= 3.13.5 OS= Linux-5.15.0-191-generic-x86_64-with-glibc2.35 [PACKAGE] numpy 2.2.6 [PACKAGE] scipy 1.16.0 [PACKAGE] pandas 2.3.1 [PACKAGE] sklearn 1.7.1 [PACKAGE] joblib 1.5.1 [PACKAGE] torch 2.7.1+cu118 [ACCELERATOR] cuda= True device= Tesla V100-SXM2-32GB [FOLD 1] canonical=14 structured_missing=0 [FOLD 2] canonical=14 structured_missing=0 [FOLD 3] canonical=14 structured_missing=0 [FOLD 4] canonical=14 structured_missing=0 [FOLD 5] canonical=14 structured_missing=0 [FOLD 6] canonical=14 structured_missing=0 [FOLD 7] canonical=14 structured_missing=0 [SUITE28 PREFLIGHT] BASIC PATH/DEPENDENCY CHECK PASSED [NOTE] For schema, row alignment, grade and student-context checks use train_suite28_drrm.py --preflight-only [PRECHECK] auditing student-semester context invariance across all historical folds... [CONTEXT AUDIT] F1 candidate=24 invariant=24 groups=39771 unstable={} [CONTEXT AUDIT] F2 candidate=24 invariant=24 groups=47400 unstable={} [CONTEXT AUDIT] F3 candidate=24 invariant=24 groups=55104 unstable={} [CONTEXT AUDIT] F4 candidate=24 invariant=24 groups=63374 unstable={} [CONTEXT AUDIT] F5 candidate=24 invariant=24 groups=71781 unstable={} [CONTEXT AUDIT] F6 candidate=24 invariant=24 groups=80750 unstable={} [CONTEXT AUDIT] F7 candidate=24 invariant=24 groups=89545 unstable={} [CONTEXT OWNERSHIP] frozen 24 fields: ['prev_semester_NHHK', 'prev_semester_ordinal', 'prev_passed_credits', 'prev_failed_credits', 'prev_outstanding_failed_credits', 'prev_semester_gpa10', 'prev_cumulative_gpa10', 'hist_semester_count', 'hist_semester_gpa10_mean', 'hist_cumulative_gpa10_mean', 'hist_registered_credits_mean', 'recent3_semester_count', 'recent3_semester_gpa10_mean', 'hist_warning_count', 'hist_warning_rate', 'hist_warning_consecutive_max', 'hist_warning_in_last_semester', 'hist_warning_in_last2_consecutive', 'recent3_warning_count', 'hist_warning_credit_debt_count', 'hist_warning_gpa_sem_count', 'hist_warning_gpa_cum_count', 'target_semester_registered_credits', 'target_semester_difficulty_weighted_workload'] [FEATURE PROVENANCE WARNING] original 136 common upstream features cannot be fully provenance-certified from these caches. Suite28 rebuilds course outcome aggregates as-of and never passes query grade or failure to inference. [ENV] Python=3.13.5 torch=2.7.1+cu118 device=cuda faculties=14 folds=[1, 2, 3, 4, 5, 6, 7] [SCIENTIFIC CONTRACT] B primary=DRRM; A=compensatory MIRT; contextual-only is RETRAINED; all learn without Committee input. [HOLDOUT STATUS] Historical F6/F7 have informed prior research and are not fresh prospective confirmation. [GRADES] Consistent ordinal [<5,5-6,6-7,7-8,8-9,9-10]; failure equals first bin. No same-semester outcomes before prediction. ================================================================================================================================== [DEVELOPMENT ORIGIN] F4 — independent earlier-only fitting ================================================================================================================================== [EVENT IDENTITY AUDIT] F4 coarse student-course-semester duplicate rows=0 / 376999 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F4 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F4 train_rows=376,999 target_rows=41,376 contexts=24 common_raw_full=136 FOLD 4 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 4 khoa_cong_nghe_thong_tin 5072 244 4828 0.048107 0.997831 0.958399 True 4 khoa_quan_tri_kinh_doanh_va_du_lich 6440 241 6199 0.037422 0.997205 0.973137 True 4 khoa_quoc_te_hoc 2202 35 2167 0.015895 0.992734 0.915077 True 4 khoa_tieng_anh 6636 68 6568 0.010247 0.972875 0.880952 True 4 khoa_tieng_bo_dao_nha 869 64 805 0.073648 0.998849 0.926352 True 4 khoa_tieng_duc 1746 44 1702 0.025200 0.995418 0.939290 True 4 khoa_tieng_han_quoc 3351 38 3313 0.011340 0.979409 0.911071 True 4 khoa_tieng_italia 1795 34 1761 0.018942 1.000000 0.813928 True 4 khoa_tieng_nga 1659 114 1545 0.068716 0.998192 0.793852 True 4 khoa_tieng_nhat 2413 38 2375 0.015748 0.998342 0.928305 True 4 khoa_tieng_phap 2275 19 2256 0.008352 1.000000 0.835604 True 4 khoa_tieng_tay_ban_nha 1306 39 1267 0.029862 1.000000 0.876723 True 4 khoa_tieng_trung_quoc 5107 104 5003 0.020364 0.967104 0.799491 True 4 khoa_viet_nam_hoc 505 18 487 0.035644 1.000000 0.853465 True [SELECT] fold=4 model=suite28_context_only epoch=1/7 loss=1.68887 AP=0.22065 F2@20=0.30645 steps=345 sec=43.1 gradients={'context': 0.03820476308465004, 'course_meta': 0.014746826142072678, 'course_id': 0.008847350254654884, 'retention': 0.002036216901615262, 'grade_threshold': 0.10496251285076141} [SELECT] fold=4 model=suite28_context_only epoch=2/7 loss=1.57166 AP=0.23700 F2@20=0.31235 steps=345 sec=39.6 gradients={'context': 0.07543907314538956, 'course_meta': 0.03676185756921768, 'course_id': 0.012360394932329655, 'retention': 0.009398327209055424, 'grade_threshold': 0.09321878105401993} [SELECT] fold=4 model=suite28_context_only epoch=3/7 loss=1.50115 AP=0.21630 F2@20=0.29451 steps=345 sec=39.4 gradients={'context': 0.16350480914115906, 'course_meta': 0.17499570548534393, 'course_id': 0.027824988588690758, 'retention': 0.002497286768630147, 'grade_threshold': 0.05626339837908745} [SELECT] fold=4 model=suite28_context_only epoch=4/7 loss=1.44985 AP=0.19148 F2@20=0.28486 steps=345 sec=38.1 gradients={'context': 0.06471525877714157, 'course_meta': 0.1544038951396942, 'course_id': 0.025115594267845154, 'retention': 0.00559823727235198, 'grade_threshold': 0.09169627726078033} [SELECT] fold=4 model=suite28_context_only epoch=5/7 loss=1.41834 AP=0.19857 F2@20=0.29159 steps=345 sec=38.1 gradients={'context': 0.08347224444150925, 'course_meta': 0.09378142654895782, 'course_id': 0.022859180346131325, 'retention': 0.003614132758229971, 'grade_threshold': 0.09168360382318497} [SELECT] fold=4 model=suite28_context_only epoch=6/7 loss=1.39315 AP=0.17780 F2@20=0.27766 steps=345 sec=37.3 gradients={'context': 0.1396232396364212, 'course_meta': 0.2059973031282425, 'course_id': 0.020136822015047073, 'retention': 0.0025880341418087482, 'grade_threshold': 0.09492738544940948} [SELECT] fold=4 model=suite28_context_only epoch=7/7 loss=1.37335 AP=0.18499 F2@20=0.27837 steps=345 sec=36.7 gradients={'context': 0.13523995876312256, 'course_meta': 0.3327379524707794, 'course_id': 0.021899692714214325, 'retention': 0.02301395684480667, 'grade_threshold': 0.11978896707296371} [SELECT] fold=4 model=suite28_mirt epoch=1/7 loss=1.67377 AP=0.32277 F2@20=0.33445 steps=345 sec=52.8 gradients={'context': 0.026622630655765533, 'course_meta': 0.016985302790999413, 'course_id': 0.017083534970879555, 'retention': 0.00023134914226830006, 'grade_threshold': 0.09476115554571152} [SELECT] fold=4 model=suite28_mirt epoch=2/7 loss=1.55311 AP=0.30508 F2@20=0.32386 steps=345 sec=50.4 gradients={'context': 0.06960230320692062, 'course_meta': 0.03639428690075874, 'course_id': 0.020772498100996017, 'retention': 0.0006316878134384751, 'grade_threshold': 0.08729688078165054} [SELECT] fold=4 model=suite28_mirt epoch=3/7 loss=1.47999 AP=0.28046 F2@20=0.30698 steps=345 sec=57.6 gradients={'context': 0.2074842005968094, 'course_meta': 0.16054511070251465, 'course_id': 0.04457956925034523, 'retention': 0.006332435179501772, 'grade_threshold': 0.0432872548699379} [SELECT] fold=4 model=suite28_mirt epoch=4/7 loss=1.42153 AP=0.27297 F2@20=0.30198 steps=345 sec=57.5 gradients={'context': 0.0552414134144783, 'course_meta': 0.22876061499118805, 'course_id': 0.053624920547008514, 'retention': 0.009520506486296654, 'grade_threshold': 0.08284422755241394} [SELECT] fold=4 model=suite28_mirt epoch=5/7 loss=1.38442 AP=0.27779 F2@20=0.30483 steps=345 sec=51.1 gradients={'context': 0.1042824387550354, 'course_meta': 0.16509388387203217, 'course_id': 0.04665352404117584, 'retention': 0.0007748891366645694, 'grade_threshold': 0.08585067838430405} [SELECT] fold=4 model=suite28_mirt epoch=6/7 loss=1.35808 AP=0.27202 F2@20=0.30210 steps=345 sec=54.2 gradients={'context': 0.1536247581243515, 'course_meta': 0.22007638216018677, 'course_id': 0.0446716770529747, 'retention': 0.0017287579830735922, 'grade_threshold': 0.07260242849588394} [SELECT] fold=4 model=suite28_mirt epoch=7/7 loss=1.33720 AP=0.27544 F2@20=0.29960 steps=345 sec=50.8 gradients={'context': 0.17202000319957733, 'course_meta': 0.39158108830451965, 'course_id': 0.05106668546795845, 'retention': 0.006239731330424547, 'grade_threshold': 0.09832920134067535} [SELECT] fold=4 model=suite28_drrm epoch=1/7 loss=1.67611 AP=0.30371 F2@20=0.32807 steps=345 sec=49.0 gradients={'context': 0.03013065829873085, 'course_meta': 0.014742759056389332, 'course_id': 0.009651504456996918, 'retention': 0.0005352086154744029, 'grade_threshold': 0.09740674495697021} [SELECT] fold=4 model=suite28_drrm epoch=2/7 loss=1.55891 AP=0.31422 F2@20=0.33031 steps=345 sec=66.4 gradients={'context': 0.04528411105275154, 'course_meta': 0.03830093517899513, 'course_id': 0.013312093913555145, 'retention': 0.0015327365836128592, 'grade_threshold': 0.08713187277317047} [SELECT] fold=4 model=suite28_drrm epoch=3/7 loss=1.48526 AP=0.30152 F2@20=0.32038 steps=345 sec=67.7 gradients={'context': 0.15645872056484222, 'course_meta': 0.1648745834827423, 'course_id': 0.029047280550003052, 'retention': 0.008119705133140087, 'grade_threshold': 0.04444977641105652} [SELECT] fold=4 model=suite28_drrm epoch=4/7 loss=1.43238 AP=0.29401 F2@20=0.31481 steps=345 sec=73.6 gradients={'context': 0.057302385568618774, 'course_meta': 0.16613265872001648, 'course_id': 0.02785765379667282, 'retention': 0.0048677027225494385, 'grade_threshold': 0.08205153793096542} [SELECT] fold=4 model=suite28_drrm epoch=5/7 loss=1.39942 AP=0.29324 F2@20=0.31755 steps=345 sec=67.8 gradients={'context': 0.07164052128791809, 'course_meta': 0.1147707924246788, 'course_id': 0.023757757619023323, 'retention': 0.002123590325936675, 'grade_threshold': 0.08718650043010712} [SELECT] fold=4 model=suite28_drrm epoch=6/7 loss=1.37459 AP=0.28667 F2@20=0.31516 steps=345 sec=67.7 gradients={'context': 0.10866496711969376, 'course_meta': 0.18127453327178955, 'course_id': 0.018665073439478874, 'retention': 0.0023609958589076996, 'grade_threshold': 0.07320025563240051} [SELECT] fold=4 model=suite28_drrm epoch=7/7 loss=1.35444 AP=0.28710 F2@20=0.31614 steps=345 sec=65.7 gradients={'context': 0.11986450105905533, 'course_meta': 0.3078349232673645, 'course_id': 0.021717188879847527, 'retention': 0.0050797415897250175, 'grade_threshold': 0.09456882625818253} ================================================================================================================================== [DEVELOPMENT ORIGIN] F5 — independent earlier-only fitting ================================================================================================================================== [EVENT IDENTITY AUDIT] F5 coarse student-course-semester duplicate rows=0 / 419467 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F5 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F5 train_rows=419,467 target_rows=50,698 contexts=24 common_raw_full=136 FOLD 5 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 5 khoa_cong_nghe_thong_tin 6398 351 6047 0.054861 0.702094 0.940919 True 5 khoa_quan_tri_kinh_doanh_va_du_lich 7604 407 7197 0.053524 0.684245 0.976328 True 5 khoa_quoc_te_hoc 2674 58 2616 0.021690 0.620045 0.923336 True 5 khoa_tieng_anh 8345 132 8213 0.015818 0.840144 0.770042 True 5 khoa_tieng_bo_dao_nha 1138 80 1058 0.070299 0.755712 0.915641 True 5 khoa_tieng_duc 1975 59 1916 0.029873 0.652152 0.879494 True 5 khoa_tieng_han_quoc 3573 67 3506 0.018752 0.728519 0.934229 True 5 khoa_tieng_italia 2129 83 2046 0.038985 0.764678 0.818694 True 5 khoa_tieng_nga 2047 134 1913 0.065462 0.718124 0.891549 True 5 khoa_tieng_nhat 2593 69 2524 0.026610 0.711145 0.810644 True 5 khoa_tieng_phap 2983 57 2926 0.019108 0.709018 0.908146 True 5 khoa_tieng_tay_ban_nha 1606 84 1522 0.052304 0.684932 0.843088 True 5 khoa_tieng_trung_quoc 6995 155 6840 0.022159 0.815868 0.732809 True 5 khoa_viet_nam_hoc 638 61 577 0.095611 0.457680 0.877743 True [SELECT] fold=5 model=suite28_context_only epoch=1/7 loss=1.68887 AP=0.20092 F2@20=0.32951 steps=349 sec=48.2 gradients={'context': 0.16102084517478943, 'course_meta': 0.0634649395942688, 'course_id': 0.02098125033080578, 'retention': 0.004242064896970987, 'grade_threshold': 0.19433559477329254} [SELECT] fold=5 model=suite28_context_only epoch=2/7 loss=1.56635 AP=0.19016 F2@20=0.31089 steps=349 sec=48.4 gradients={'context': 0.06795458495616913, 'course_meta': 0.10278412699699402, 'course_id': 0.01266526896506548, 'retention': 0.004230151418596506, 'grade_threshold': 0.15023908019065857} [SELECT] fold=5 model=suite28_context_only epoch=3/7 loss=1.49592 AP=0.19387 F2@20=0.30947 steps=349 sec=50.1 gradients={'context': 0.087455153465271, 'course_meta': 0.05783785507082939, 'course_id': 0.0192087572067976, 'retention': 0.00019752781372517347, 'grade_threshold': 0.13336879014968872} [SELECT] fold=5 model=suite28_context_only epoch=4/7 loss=1.44620 AP=0.18808 F2@20=0.30929 steps=349 sec=48.4 gradients={'context': 0.05386853963136673, 'course_meta': 0.14108514785766602, 'course_id': 0.02479100599884987, 'retention': 0.005402527283877134, 'grade_threshold': 0.0663684606552124} [SELECT] fold=5 model=suite28_context_only epoch=5/7 loss=1.41199 AP=0.17979 F2@20=0.29559 steps=349 sec=49.3 gradients={'context': 0.1050070971250534, 'course_meta': 0.1106330156326294, 'course_id': 0.02615470625460148, 'retention': 0.016113243997097015, 'grade_threshold': 0.1675291508436203} [SELECT] fold=5 model=suite28_context_only epoch=6/7 loss=1.38763 AP=0.17378 F2@20=0.29389 steps=349 sec=47.8 gradients={'context': 0.12091000378131866, 'course_meta': 0.18743428587913513, 'course_id': 0.026703989133238792, 'retention': 0.012538924813270569, 'grade_threshold': 0.1484573930501938} [SELECT] fold=5 model=suite28_context_only epoch=7/7 loss=1.36993 AP=0.17286 F2@20=0.28870 steps=349 sec=48.9 gradients={'context': 0.0955527275800705, 'course_meta': 0.12929728627204895, 'course_id': 0.022479794919490814, 'retention': 0.014643063768744469, 'grade_threshold': 0.08552031219005585} [SELECT] fold=5 model=suite28_mirt epoch=1/7 loss=1.67318 AP=0.23627 F2@20=0.34855 steps=349 sec=72.9 gradients={'context': 0.06746077537536621, 'course_meta': 0.03988170996308327, 'course_id': 0.04172550514340401, 'retention': 0.004403670784085989, 'grade_threshold': 0.1701575368642807} [SELECT] fold=5 model=suite28_mirt epoch=2/7 loss=1.54650 AP=0.22294 F2@20=0.32874 steps=349 sec=72.0 gradients={'context': 0.05688929185271263, 'course_meta': 0.10505992919206619, 'course_id': 0.021982466802001, 'retention': 0.0011099509429186583, 'grade_threshold': 0.13771292567253113} [SELECT] fold=5 model=suite28_mirt epoch=3/7 loss=1.47109 AP=0.22784 F2@20=0.32576 steps=349 sec=72.4 gradients={'context': 0.06054283678531647, 'course_meta': 0.09106834977865219, 'course_id': 0.03321104124188423, 'retention': 2.767855767160654e-05, 'grade_threshold': 0.1374385952949524} [SELECT] fold=5 model=suite28_mirt epoch=4/7 loss=1.41368 AP=0.22063 F2@20=0.32155 steps=349 sec=62.9 gradients={'context': 0.05670991539955139, 'course_meta': 0.19070647656917572, 'course_id': 0.05023299157619476, 'retention': 0.0007485027308575809, 'grade_threshold': 0.06842666119337082} [SELECT] fold=5 model=suite28_mirt epoch=5/7 loss=1.37546 AP=0.22147 F2@20=0.32847 steps=349 sec=58.0 gradients={'context': 0.11207699775695801, 'course_meta': 0.140860453248024, 'course_id': 0.06241369992494583, 'retention': 0.004736221395432949, 'grade_threshold': 0.13685311377048492} [SELECT] fold=5 model=suite28_mirt epoch=6/7 loss=1.34861 AP=0.22716 F2@20=0.32987 steps=349 sec=57.2 gradients={'context': 0.09772969782352448, 'course_meta': 0.1742594987154007, 'course_id': 0.06509402394294739, 'retention': 0.0025269025936722755, 'grade_threshold': 0.08307815343141556} [SELECT] fold=5 model=suite28_mirt epoch=7/7 loss=1.33160 AP=0.21695 F2@20=0.32794 steps=349 sec=56.5 gradients={'context': 0.08161941170692444, 'course_meta': 0.1999630331993103, 'course_id': 0.05311162397265434, 'retention': 0.011907495558261871, 'grade_threshold': 0.058996785432100296} [SELECT] fold=5 model=suite28_drrm epoch=1/7 loss=1.67541 AP=0.24124 F2@20=0.34770 steps=349 sec=58.6 gradients={'context': 0.046225130558013916, 'course_meta': 0.03940427303314209, 'course_id': 0.023852020502090454, 'retention': 0.007697949185967445, 'grade_threshold': 0.17365820705890656} [SELECT] fold=5 model=suite28_drrm epoch=2/7 loss=1.55175 AP=0.22848 F2@20=0.32876 steps=349 sec=56.1 gradients={'context': 0.050557926297187805, 'course_meta': 0.12136776000261307, 'course_id': 0.014255724847316742, 'retention': 0.0017872703028842807, 'grade_threshold': 0.13981692492961884} [SELECT] fold=5 model=suite28_drrm epoch=3/7 loss=1.47771 AP=0.22774 F2@20=0.33384 steps=349 sec=58.2 gradients={'context': 0.05336727574467659, 'course_meta': 0.06878522783517838, 'course_id': 0.020728617906570435, 'retention': 1.374678686261177e-05, 'grade_threshold': 0.1234787106513977} [SELECT] fold=5 model=suite28_drrm epoch=4/7 loss=1.42643 AP=0.22793 F2@20=0.33086 steps=349 sec=57.2 gradients={'context': 0.04239115118980408, 'course_meta': 0.14703668653964996, 'course_id': 0.028267117217183113, 'retention': 0.0005461079999804497, 'grade_threshold': 0.06415882706642151} [SELECT] fold=5 model=suite28_drrm epoch=5/7 loss=1.39129 AP=0.22010 F2@20=0.32868 steps=349 sec=57.4 gradients={'context': 0.0715532898902893, 'course_meta': 0.10650026798248291, 'course_id': 0.02911658026278019, 'retention': 0.0009158237371593714, 'grade_threshold': 0.12717516720294952} [SELECT] fold=5 model=suite28_drrm epoch=6/7 loss=1.36616 AP=0.22753 F2@20=0.33105 steps=349 sec=60.0 gradients={'context': 0.08039433509111404, 'course_meta': 0.16977132856845856, 'course_id': 0.030013395473361015, 'retention': 0.013941967859864235, 'grade_threshold': 0.09613308310508728} [SELECT] fold=5 model=suite28_drrm epoch=7/7 loss=1.34957 AP=0.22165 F2@20=0.32514 steps=349 sec=54.1 gradients={'context': 0.05734943225979805, 'course_meta': 0.13218621909618378, 'course_id': 0.02207782492041588, 'retention': 0.013069689273834229, 'grade_threshold': 0.06116001680493355} [DEVELOPMENT FROZEN CHOICE] suite28_context_only -------------------------------------------------------------------------------- epoch AP F2 origins 2 0.213582 0.311621 2 1 0.210785 0.317981 2 3 0.205083 0.301993 2 4 0.189781 0.297075 2 5 0.189183 0.293590 2 7 0.178927 0.283532 2 6 0.175789 0.285774 2 [DEVELOPMENT FROZEN CHOICE] suite28_mirt -------------------------------------------------------------------------------- epoch AP F2 origins 1 0.279519 0.341504 2 2 0.264006 0.326302 2 3 0.254149 0.316369 2 5 0.249631 0.316649 2 6 0.249589 0.315987 2 4 0.246801 0.311768 2 7 0.246195 0.313773 2 [DEVELOPMENT FROZEN CHOICE] suite28_drrm -------------------------------------------------------------------------------- epoch AP F2 origins 1 0.272479 0.337881 2 2 0.271347 0.329537 2 3 0.264628 0.327111 2 4 0.260974 0.322835 2 6 0.257102 0.323106 2 5 0.256670 0.323114 2 7 0.254377 0.320641 2 [FROZEN DEV EPOCHS] {'suite28_context_only': 2, 'suite28_mirt': 1, 'suite28_drrm': 1} ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F1 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F1 coarse student-course-semester duplicate rows=0 / 240672 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F1 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F1 train_rows=240,672 target_rows=45,800 contexts=24 common_raw_full=136 FOLD 1 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 1 khoa_cong_nghe_thong_tin 5222 189 5033 0.036193 0.624473 0.905592 True 1 khoa_quan_tri_kinh_doanh_va_du_lich 5672 219 5453 0.038611 0.608604 0.939880 True 1 khoa_quoc_te_hoc 2363 31 2332 0.013119 0.584427 0.884892 True 1 khoa_tieng_anh 8164 148 8016 0.018128 0.644292 0.971338 True 1 khoa_tieng_bo_dao_nha 768 30 738 0.039062 0.636719 0.984375 True 1 khoa_tieng_duc 1900 46 1854 0.024211 0.650526 0.987895 True 1 khoa_tieng_han_quoc 2904 20 2884 0.006887 0.677342 0.876033 True 1 khoa_tieng_italia 2166 111 2055 0.051247 0.761773 0.956602 True 1 khoa_tieng_nga 1588 102 1486 0.064232 0.592569 0.988665 True 1 khoa_tieng_nhat 2840 17 2823 0.005986 0.612676 0.963028 True 1 khoa_tieng_phap 2606 42 2564 0.016117 0.692249 0.990790 True 1 khoa_tieng_tay_ban_nha 1401 63 1338 0.044968 0.706638 0.986438 True 1 khoa_tieng_trung_quoc 7393 35 7358 0.004734 0.766130 0.934127 True 1 khoa_viet_nam_hoc 813 30 783 0.036900 0.548585 0.997540 True [FULL FEATURE CONTROL] F1 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.27194 secs=46.5 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F1 suite28_context_only epoch=1/2 loss=1.70348 steps=216 sec=23.2 gradients={'context': 0.0744829773902893, 'course_meta': 0.006410890258848667, 'course_id': 0.006498088128864765, 'retention': 0.013070951215922832, 'grade_threshold': 0.06340257823467255} [TRAIN] F1 suite28_context_only epoch=2/2 loss=1.60655 steps=216 sec=22.8 gradients={'context': 0.030428094789385796, 'course_meta': 0.012019466608762741, 'course_id': 0.006898149847984314, 'retention': 0.009534106589853764, 'grade_threshold': 0.0711519792675972} [PREDICTION] F1 suite28_context_only macro AP=0.18367 AUC=0.82202 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F1 suite28_mirt epoch=1/1 loss=1.69249 steps=216 sec=38.1 gradients={'context': 0.04735320061445236, 'course_meta': 0.006144833285361528, 'course_id': 0.012812460772693157, 'retention': 0.0019451930420473218, 'grade_threshold': 0.062409888952970505} [PREDICTION] F1 suite28_mirt macro AP=0.24720 AUC=0.84627 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F1 suite28_drrm epoch=1/1 loss=1.69094 steps=216 sec=30.7 gradients={'context': 0.05155457928776741, 'course_meta': 0.006734305992722511, 'course_id': 0.0067952461540699005, 'retention': 0.007016572169959545, 'grade_threshold': 0.061899732798337936} [GRADE LIKELIHOOD] observed_grade_rows= 45800 NLL= 1.687869668006897 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F1 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 5222 189 10.590961 0.123463 1.885391 0.684183 0.493708 khoa_quan_tri_kinh_doanh_va_du_lich 5672 219 9.317701 0.144278 1.949709 0.861676 0.485705 khoa_quoc_te_hoc 2363 31 8.955142 0.124946 1.887321 0.747705 0.461827 khoa_tieng_anh 8164 148 11.781725 0.120649 1.875877 0.648723 0.509450 khoa_tieng_bo_dao_nha 768 30 11.815104 0.166152 2.131020 1.291483 0.503409 khoa_tieng_duc 1900 46 12.001579 0.120178 1.873849 0.745515 0.514418 khoa_tieng_han_quoc 2904 20 12.967976 0.095783 1.673830 0.762334 0.535622 khoa_tieng_italia 2166 111 14.038781 0.149022 2.056250 1.104305 0.602351 khoa_tieng_nga 1588 102 12.210958 0.153715 2.048249 1.128749 0.470343 khoa_tieng_nhat 2840 17 12.124648 0.100757 1.691653 0.782990 0.483469 khoa_tieng_phap 2606 42 12.996930 0.120964 1.859390 0.738919 0.547313 khoa_tieng_tay_ban_nha 1401 63 14.095646 0.150446 2.082330 1.116616 0.558371 khoa_tieng_trung_quoc 7393 35 14.513324 0.101008 1.812705 0.610134 0.605810 khoa_viet_nam_hoc 813 30 7.981550 0.161657 1.961487 1.067518 0.433775 [PREDICTION] F1 suite28_drrm macro AP=0.22726 AUC=0.83469 F1 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 1 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.906000 336.900391 -2.783521 0.149252 0.117606 1.094986 0.001444 3.481850e-14 0.0 0.0 1 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.940000 611.354004 -2.715494 0.185220 0.135073 1.074634 0.001662 4.355915e-14 0.0 0.0 1 khoa_quoc_te_hoc bottleneck 2363 course_requirement 0.884892 222.421082 -2.755249 0.143288 0.110603 1.038696 0.001489 3.160375e-14 0.0 0.0 1 khoa_tieng_anh bottleneck 5000 course_requirement 0.966800 479.355011 -2.874679 0.152038 0.119835 1.117240 0.000820 3.740501e-14 0.0 0.0 1 khoa_tieng_bo_dao_nha bottleneck 768 course_requirement 0.980469 79.532555 -2.859370 0.197868 0.104732 1.093759 0.002303 3.984954e-14 0.0 0.0 1 khoa_tieng_duc bottleneck 1900 course_requirement 0.987895 185.258942 -2.792530 0.166057 0.105107 1.072410 0.001178 5.022678e-14 0.0 0.0 1 khoa_tieng_han_quoc bottleneck 2904 course_requirement 0.876033 189.516876 -2.857440 0.162770 0.116200 1.053503 0.001954 4.151489e-14 0.0 0.0 1 khoa_tieng_italia bottleneck 2166 course_requirement 0.956602 128.312103 -2.813766 0.166120 0.116095 1.095687 0.001504 4.554261e-14 0.0 0.0 1 khoa_tieng_nga bottleneck 1588 course_requirement 0.964106 119.650505 -2.804896 0.194854 0.116335 1.084580 0.002018 2.847888e-14 0.0 0.0 1 khoa_tieng_nhat bottleneck 2840 course_requirement 0.960211 293.483795 -2.884397 0.171778 0.127230 1.076409 0.001262 4.260789e-14 0.0 0.0 1 khoa_tieng_phap bottleneck 2606 course_requirement 0.981965 165.192245 -2.872097 0.225663 0.118132 1.058703 0.001922 3.992278e-14 0.0 0.0 1 khoa_tieng_tay_ban_nha bottleneck 1401 course_requirement 0.983583 122.233406 -2.900601 0.169242 0.114552 1.104582 0.001318 6.938905e-14 0.0 0.0 1 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.934000 459.796814 -2.924716 0.178490 0.120431 1.058487 0.001658 4.011166e-14 0.0 0.0 1 khoa_viet_nam_hoc bottleneck 813 course_requirement 0.997540 132.483398 -2.881825 0.143211 0.120312 1.172310 0.000482 5.291741e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 1, 'rho_base': 0.8054543137550354, 'process_noise_diag': [0.018238898366689682, 0.01857113651931286, 0.018262647092342377, 0.01863781362771988], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.8690415620803833, 0.8729681372642517, 0.887276291847229, 0.8972602486610413], [0.8756463527679443, 0.8799670338630676, 0.883378267288208, 0.8824530839920044], [0.8701438903808594, 0.8838558197021484, 0.8974323868751526, 0.9034051895141602], [0.8079217076301575, 0.8494812250137329, 0.8919417858123779, 0.8997189402580261], [0.8660005331039429, 0.8731712102890015, 0.8847633600234985, 0.8844791650772095], [0.8592850565910339, 0.8777420520782471, 0.8863818645477295, 0.8881337642669678], [0.8363313674926758, 0.8605061769485474, 0.8959755301475525, 0.9007239937782288], [0.8683971762657166, 0.8790405988693237, 0.8886831998825073, 0.8955835700035095], [0.873099684715271, 0.8784300088882446, 0.8848792910575867, 0.889029324054718], [0.8255569934844971, 0.861310601234436, 0.8873374462127686, 0.9003281593322754], [0.8498769998550415, 0.8757960200309753, 0.8963777422904968, 0.8986137509346008], [0.8446522951126099, 0.8691437840461731, 0.8859765529632568, 0.8900866508483887], [0.7989807724952698, 0.8454034328460693, 0.889083981513977, 0.9017407894134521], [0.8381154537200928, 0.8527560234069824, 0.8806699514389038, 0.8925400376319885]]}] [INTERVENTION] F1 no_history mean_delta_ap=-0.05046 [INTERVENTION] F1 no_grade mean_delta_ap=+0.01016 [INTERVENTION] F1 mean_only mean_delta_ap=+0.00642 [INTERVENTION] F1 no_course_id mean_delta_ap=-0.00044 [INTERVENTION] F1 shuffle_historical_outcomes mean_delta_ap=-0.05437 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F2 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F2 coarse student-course-semester duplicate rows=0 / 286472 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F2 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F2 train_rows=286,472 target_rows=42,122 contexts=24 common_raw_full=136 FOLD 2 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 2 khoa_cong_nghe_thong_tin 4598 152 4446 0.033058 0.999130 0.909091 True 2 khoa_quan_tri_kinh_doanh_va_du_lich 5530 166 5364 0.030018 0.994213 0.991863 True 2 khoa_quoc_te_hoc 2095 19 2076 0.009069 0.997613 0.905489 True 2 khoa_tieng_anh 8328 122 8206 0.014649 0.910183 0.763088 True 2 khoa_tieng_bo_dao_nha 716 30 686 0.041899 1.000000 0.990223 True 2 khoa_tieng_duc 1893 53 1840 0.027998 0.994717 0.991020 True 2 khoa_tieng_han_quoc 3337 29 3308 0.008690 0.990411 0.916392 True 2 khoa_tieng_italia 2024 71 1953 0.035079 0.990119 0.982708 True 2 khoa_tieng_nga 1567 83 1484 0.052967 0.997447 0.991704 True 2 khoa_tieng_nhat 2448 11 2437 0.004493 0.998366 0.988562 True 2 khoa_tieng_phap 2204 40 2164 0.018149 0.997278 0.926951 True 2 khoa_tieng_tay_ban_nha 1242 42 1200 0.033816 1.000000 0.889694 True 2 khoa_tieng_trung_quoc 5584 76 5508 0.013610 0.890938 0.831483 True 2 khoa_viet_nam_hoc 556 39 517 0.070144 1.000000 0.996403 True [FULL FEATURE CONTROL] F2 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.37410 secs=17.1 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F2 suite28_context_only epoch=1/2 loss=1.69257 steps=274 sec=20.3 gradients={'context': 0.06896419078111649, 'course_meta': 0.011082264594733715, 'course_id': 0.006337997503578663, 'retention': 0.01255648024380207, 'grade_threshold': 0.08905555307865143} [TRAIN] F2 suite28_context_only epoch=2/2 loss=1.59145 steps=274 sec=20.9 gradients={'context': 0.07059818506240845, 'course_meta': 0.0468868650496006, 'course_id': 0.008682901039719582, 'retention': 0.015691041946411133, 'grade_threshold': 0.06355208903551102} [PREDICTION] F2 suite28_context_only macro AP=0.17406 AUC=0.81167 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F2 suite28_mirt epoch=1/1 loss=1.68055 steps=274 sec=30.6 gradients={'context': 0.03888310119509697, 'course_meta': 0.008026102557778358, 'course_id': 0.012232952751219273, 'retention': 0.003960205242037773, 'grade_threshold': 0.08733033388853073} [PREDICTION] F2 suite28_mirt macro AP=0.27296 AUC=0.85144 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F2 suite28_drrm epoch=1/1 loss=1.68172 steps=274 sec=27.1 gradients={'context': 0.047215547412633896, 'course_meta': 0.007575878407806158, 'course_id': 0.006610232871025801, 'retention': 0.008450559340417385, 'grade_threshold': 0.08818008750677109} [GRADE LIKELIHOOD] observed_grade_rows= 42122 NLL= 1.6243597269058228 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F2 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 4598 152 14.893432 0.136729 1.894737 1.007338 0.800967 khoa_quan_tri_kinh_doanh_va_du_lich 5530 166 12.368355 0.166647 2.017443 1.239351 0.796249 khoa_quoc_te_hoc 2095 19 13.170405 0.141601 1.934643 1.124165 0.799824 khoa_tieng_anh 8328 122 16.466619 0.134669 1.791413 0.926327 0.727381 khoa_tieng_bo_dao_nha 716 30 17.100559 0.178648 2.120413 1.574871 0.801861 khoa_tieng_duc 1893 53 18.245113 0.135098 1.837481 0.983949 0.796231 khoa_tieng_han_quoc 3337 29 17.895714 0.095424 1.505400 0.743783 0.791984 khoa_tieng_italia 2024 71 18.977766 0.158156 2.039427 1.409820 0.793784 khoa_tieng_nga 1567 83 16.495852 0.176419 2.053998 1.467282 0.799648 khoa_tieng_nhat 2448 11 18.689135 0.102347 1.523047 0.775931 0.800490 khoa_tieng_phap 2204 40 16.133394 0.132762 1.841026 1.046686 0.799693 khoa_tieng_tay_ban_nha 1242 42 18.611916 0.150896 1.988555 1.373127 0.801837 khoa_tieng_trung_quoc 5584 76 15.329692 0.109727 1.602350 0.679190 0.714436 khoa_viet_nam_hoc 556 39 13.181655 0.164014 1.923669 1.247914 0.801940 [PREDICTION] F2 suite28_drrm macro AP=0.25085 AUC=0.85423 F2 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 2 khoa_cong_nghe_thong_tin bottleneck 4598 course_requirement 0.908438 325.822968 -2.788055 0.248646 0.145768 1.039946 0.006374 4.016474e-14 0.0 0.0 2 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.988600 649.676575 -2.840466 0.419582 0.163773 1.011692 0.020481 3.487377e-14 0.0 0.0 2 khoa_quoc_te_hoc bottleneck 2095 course_requirement 0.905489 263.426727 -2.843524 0.363110 0.138370 1.019384 0.010743 3.410754e-14 0.0 0.0 2 khoa_tieng_anh bottleneck 5000 course_requirement 0.763400 345.613800 -3.008329 0.381575 0.124621 1.017770 0.020745 5.013328e-14 0.0 0.0 2 khoa_tieng_bo_dao_nha bottleneck 716 course_requirement 0.981844 84.935753 -2.956694 0.427418 0.129254 1.033656 0.014979 3.022826e-14 0.0 0.0 2 khoa_tieng_duc bottleneck 1893 course_requirement 0.990491 193.436874 -2.867881 0.257178 0.146111 1.048559 0.005619 3.937558e-14 0.0 0.0 2 khoa_tieng_han_quoc bottleneck 3337 course_requirement 0.916392 216.954742 -2.954596 0.275267 0.118824 1.050486 0.007020 6.474663e-14 0.0 0.0 2 khoa_tieng_italia bottleneck 2024 course_requirement 0.978755 132.863144 -2.864022 0.269039 0.145055 1.055057 0.006452 3.525295e-14 0.0 0.0 2 khoa_tieng_nga bottleneck 1567 course_requirement 0.966177 137.948303 -2.771696 0.237971 0.150623 1.066617 0.005451 4.324102e-14 0.0 0.0 2 khoa_tieng_nhat bottleneck 2448 course_requirement 0.986520 315.305145 -2.978393 0.249667 0.131925 1.071310 0.005304 3.720794e-14 0.0 0.0 2 khoa_tieng_phap bottleneck 2204 course_requirement 0.921053 188.160614 -3.006829 0.349384 0.133732 1.034679 0.011346 5.570526e-14 0.0 0.0 2 khoa_tieng_tay_ban_nha bottleneck 1242 course_requirement 0.884058 136.113525 -3.006645 0.253080 0.114525 1.067383 0.005189 4.168165e-14 0.0 0.0 2 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.828400 394.917206 -3.071178 0.329121 0.118995 1.035274 0.013883 5.808891e-14 0.0 0.0 2 khoa_viet_nam_hoc bottleneck 556 course_requirement 0.996403 132.570145 -3.066856 0.210926 0.120604 1.067492 0.003525 1.979895e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 2, 'rho_base': 0.8019404411315918, 'process_noise_diag': [0.017794858664274216, 0.018343118950724602, 0.017852898687124252, 0.018392829224467278], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.8905611038208008, 0.8929461240768433, 0.9081255197525024, 0.9137678146362305], [0.900750994682312, 0.9133844375610352, 0.9239439964294434, 0.923757016658783], [0.8878998756408691, 0.9005150198936462, 0.9152706265449524, 0.9168630838394165], [0.8074705004692078, 0.8703452944755554, 0.9191507697105408, 0.9261291027069092], [0.8744964003562927, 0.8864402174949646, 0.8918917775154114, 0.9006705284118652], [0.8760550022125244, 0.899625301361084, 0.9143255949020386, 0.9136082530021667], [0.8372727036476135, 0.8708491325378418, 0.9148390293121338, 0.9279897809028625], [0.8907386660575867, 0.9042389392852783, 0.9095369577407837, 0.9100215435028076], [0.9031748175621033, 0.9128234386444092, 0.9138518571853638, 0.9136790633201599], [0.8371761441230774, 0.8816936612129211, 0.9131449460983276, 0.9267982244491577], [0.8663966655731201, 0.8943272233009338, 0.9143048524856567, 0.917034924030304], [0.8567942380905151, 0.8959105610847473, 0.9083256125450134, 0.9225460290908813], [0.8190476894378662, 0.8682632446289062, 0.9118741750717163, 0.9297548532485962], [0.8567718267440796, 0.8753349184989929, 0.914482593536377, 0.9224730730056763]]}] [INTERVENTION] F2 no_history mean_delta_ap=-0.06868 [INTERVENTION] F2 no_grade mean_delta_ap=+0.02883 [INTERVENTION] F2 mean_only mean_delta_ap=-0.00025 [INTERVENTION] F2 no_course_id mean_delta_ap=+0.00379 [INTERVENTION] F2 shuffle_historical_outcomes mean_delta_ap=-0.07927 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F3 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F3 coarse student-course-semester duplicate rows=0 / 329579 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F3 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F3 train_rows=329,579 target_rows=47,420 contexts=24 common_raw_full=136 FOLD 3 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 3 khoa_cong_nghe_thong_tin 6277 218 6059 0.034730 0.689023 0.957145 True 3 khoa_quan_tri_kinh_doanh_va_du_lich 6491 209 6282 0.032198 0.666307 0.906486 True 3 khoa_quoc_te_hoc 2665 26 2639 0.009756 0.646529 0.869418 True 3 khoa_tieng_anh 7279 101 7178 0.013876 0.818519 0.893117 True 3 khoa_tieng_bo_dao_nha 1032 50 982 0.048450 0.743217 0.904070 True 3 khoa_tieng_duc 1909 47 1862 0.024620 0.767941 0.898900 True 3 khoa_tieng_han_quoc 3235 42 3193 0.012983 0.721484 0.897682 True 3 khoa_tieng_italia 2145 68 2077 0.031702 0.814918 0.932401 True 3 khoa_tieng_nga 1996 92 1904 0.046092 0.766533 0.892786 True 3 khoa_tieng_nhat 2588 17 2571 0.006569 0.610124 0.933926 True 3 khoa_tieng_phap 2693 43 2650 0.015967 0.733754 0.902711 True 3 khoa_tieng_tay_ban_nha 1417 54 1363 0.038109 0.745236 0.874382 True 3 khoa_tieng_trung_quoc 6920 103 6817 0.014884 0.793931 0.851734 True 3 khoa_viet_nam_hoc 773 33 740 0.042691 0.495472 0.908150 True [FULL FEATURE CONTROL] F3 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.30472 secs=169.1 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F3 suite28_context_only epoch=1/2 loss=1.69380 steps=282 sec=29.5 gradients={'context': 0.056594330817461014, 'course_meta': 0.013753390870988369, 'course_id': 0.008966137655079365, 'retention': 0.010839094407856464, 'grade_threshold': 0.09634637087583542} [TRAIN] F3 suite28_context_only epoch=2/2 loss=1.58800 steps=282 sec=33.1 gradients={'context': 0.05972585827112198, 'course_meta': 0.023213231936097145, 'course_id': 0.013174700550734997, 'retention': 0.004463711753487587, 'grade_threshold': 0.08385858684778214} [PREDICTION] F3 suite28_context_only macro AP=0.18695 AUC=0.78726 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F3 suite28_mirt epoch=1/1 loss=1.68048 steps=282 sec=57.0 gradients={'context': 0.04892070218920708, 'course_meta': 0.013973131775856018, 'course_id': 0.01689840294420719, 'retention': 0.004292761441320181, 'grade_threshold': 0.09141338616609573} [PREDICTION] F3 suite28_mirt macro AP=0.24479 AUC=0.81107 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F3 suite28_drrm epoch=1/1 loss=1.68163 steps=282 sec=40.9 gradients={'context': 0.045754220336675644, 'course_meta': 0.014702166430652142, 'course_id': 0.009216303937137127, 'retention': 0.008515424095094204, 'grade_threshold': 0.09292846918106079} [GRADE LIKELIHOOD] observed_grade_rows= 47420 NLL= 1.6768019199371338 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F3 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 6277 218 12.221443 0.189166 2.167228 1.148564 0.544356 khoa_quan_tri_kinh_doanh_va_du_lich 6491 209 10.022031 0.232313 2.327170 1.371372 0.532004 khoa_quoc_te_hoc 2665 26 9.958349 0.182272 2.150452 1.192829 0.510821 khoa_tieng_anh 7279 101 14.547053 0.143449 1.929732 0.847145 0.646773 khoa_tieng_bo_dao_nha 1032 50 11.954457 0.233319 2.376880 1.556694 0.586828 khoa_tieng_duc 1909 47 14.064956 0.169106 2.059552 1.050151 0.605234 khoa_tieng_han_quoc 3235 42 13.955796 0.127435 1.739303 0.783754 0.568766 khoa_tieng_italia 2145 68 15.426107 0.194832 2.234366 1.384656 0.643979 khoa_tieng_nga 1996 92 12.437375 0.230560 2.336317 1.515906 0.604805 khoa_tieng_nhat 2588 17 11.062596 0.135707 1.826695 0.779629 0.479339 khoa_tieng_phap 2693 43 14.266989 0.158262 2.019772 1.032485 0.579157 khoa_tieng_tay_ban_nha 1417 54 14.214538 0.181651 2.171462 1.346868 0.588511 khoa_tieng_trung_quoc 6920 103 15.227168 0.119753 1.743516 0.620214 0.627479 khoa_viet_nam_hoc 773 33 7.613195 0.206521 2.224459 1.197062 0.391584 [PREDICTION] F3 suite28_drrm macro AP=0.23445 AUC=0.80423 F3 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 3 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.956800 466.584991 -2.747448 0.234012 0.195114 1.080001 0.003859 5.345878e-14 0.0 0.0 3 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.908200 686.250793 -2.677465 0.252103 0.199026 1.034263 0.009537 4.318954e-14 0.0 0.0 3 khoa_quoc_te_hoc bottleneck 2665 course_requirement 0.869418 267.009369 -2.752066 0.202841 0.163428 1.042683 0.005024 3.330284e-14 0.0 0.0 3 khoa_tieng_anh bottleneck 5000 course_requirement 0.888600 592.918030 -3.089752 0.605586 0.165810 1.017405 0.055425 3.851663e-14 0.0 0.0 3 khoa_tieng_bo_dao_nha bottleneck 1032 course_requirement 0.896318 88.750000 -2.914079 0.341283 0.152912 1.033603 0.028727 2.501395e-14 0.0 0.0 3 khoa_tieng_duc bottleneck 1909 course_requirement 0.894709 185.955994 -2.840950 0.461363 0.183847 1.009855 0.062111 3.531285e-14 0.0 0.0 3 khoa_tieng_han_quoc bottleneck 3235 course_requirement 0.897063 210.485626 -2.900482 0.390980 0.172006 1.016729 0.036446 3.931677e-14 0.0 0.0 3 khoa_tieng_italia bottleneck 2145 course_requirement 0.928205 153.423309 -2.851110 0.410294 0.178022 1.013088 0.051277 4.332248e-14 0.0 0.0 3 khoa_tieng_nga bottleneck 1996 course_requirement 0.888778 142.394287 -2.836569 0.507636 0.209254 1.011689 0.074181 2.341215e-14 0.0 0.0 3 khoa_tieng_nhat bottleneck 2588 course_requirement 0.929289 320.443573 -2.945744 0.309722 0.160376 1.016969 0.022937 2.997590e-14 0.0 0.0 3 khoa_tieng_phap bottleneck 2693 course_requirement 0.884515 194.832153 -2.975947 0.486498 0.173360 1.015234 0.033448 5.339781e-14 0.0 0.0 3 khoa_tieng_tay_ban_nha bottleneck 1417 course_requirement 0.871560 116.285812 -3.117540 0.631034 0.204336 1.005985 0.127771 3.345121e-14 0.0 0.0 3 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.847600 446.713409 -2.989054 0.318314 0.158067 1.015289 0.028702 3.409532e-14 0.0 0.0 3 khoa_viet_nam_hoc bottleneck 773 course_requirement 0.908150 152.553680 -2.983501 0.463444 0.181472 1.007462 0.070492 2.979719e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 3, 'rho_base': 0.8049254417419434, 'process_noise_diag': [0.017790846526622772, 0.018470177426934242, 0.018008161336183548, 0.01859026588499546], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.8945342302322388, 0.8986354470252991, 0.9080462455749512, 0.920079231262207], [0.9008962512016296, 0.909356415271759, 0.9201935529708862, 0.9166025519371033], [0.8954031467437744, 0.9071891903877258, 0.9198296070098877, 0.9150430560112], [0.7902219295501709, 0.8700686693191528, 0.928256094455719, 0.932986319065094], [0.8795698285102844, 0.8989071846008301, 0.9037013053894043, 0.9012719392776489], [0.8842139840126038, 0.9074264764785767, 0.9225003719329834, 0.9224308729171753], [0.8531318306922913, 0.8857079744338989, 0.9225181937217712, 0.9308627247810364], [0.8913765549659729, 0.9034852385520935, 0.9130771160125732, 0.9153507351875305], [0.8991084694862366, 0.913368284702301, 0.9096781015396118, 0.9129378795623779], [0.843429446220398, 0.8868969678878784, 0.9221469163894653, 0.9319100379943848], [0.8650955557823181, 0.8985022306442261, 0.9189404249191284, 0.9228049516677856], [0.852694034576416, 0.8824325799942017, 0.9096066951751709, 0.9235246777534485], [0.8177122473716736, 0.8789488077163696, 0.9231323599815369, 0.9266905784606934], [0.8501701951026917, 0.8812470436096191, 0.9086227416992188, 0.9222480058670044]]}] [INTERVENTION] F3 no_history mean_delta_ap=-0.03587 [INTERVENTION] F3 no_grade mean_delta_ap=+0.01277 [INTERVENTION] F3 mean_only mean_delta_ap=-0.00293 [INTERVENTION] F3 no_course_id mean_delta_ap=-0.00058 [INTERVENTION] F3 shuffle_historical_outcomes mean_delta_ap=-0.07344 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F4 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F4 coarse student-course-semester duplicate rows=0 / 376999 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F4 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F4 train_rows=376,999 target_rows=41,376 contexts=24 common_raw_full=136 FOLD 4 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 4 khoa_cong_nghe_thong_tin 5072 244 4828 0.048107 0.997831 0.958399 True 4 khoa_quan_tri_kinh_doanh_va_du_lich 6440 241 6199 0.037422 0.997205 0.973137 True 4 khoa_quoc_te_hoc 2202 35 2167 0.015895 0.992734 0.915077 True 4 khoa_tieng_anh 6636 68 6568 0.010247 0.972875 0.880952 True 4 khoa_tieng_bo_dao_nha 869 64 805 0.073648 0.998849 0.926352 True 4 khoa_tieng_duc 1746 44 1702 0.025200 0.995418 0.939290 True 4 khoa_tieng_han_quoc 3351 38 3313 0.011340 0.979409 0.911071 True 4 khoa_tieng_italia 1795 34 1761 0.018942 1.000000 0.813928 True 4 khoa_tieng_nga 1659 114 1545 0.068716 0.998192 0.793852 True 4 khoa_tieng_nhat 2413 38 2375 0.015748 0.998342 0.928305 True 4 khoa_tieng_phap 2275 19 2256 0.008352 1.000000 0.835604 True 4 khoa_tieng_tay_ban_nha 1306 39 1267 0.029862 1.000000 0.876723 True 4 khoa_tieng_trung_quoc 5107 104 5003 0.020364 0.967104 0.799491 True 4 khoa_viet_nam_hoc 505 18 487 0.035644 1.000000 0.853465 True [FULL FEATURE CONTROL] F4 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.35302 secs=203.8 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F4 suite28_context_only epoch=1/2 loss=1.68896 steps=345 sec=37.6 gradients={'context': 0.03625708073377609, 'course_meta': 0.022466806694865227, 'course_id': 0.011324433609843254, 'retention': 0.004176975227892399, 'grade_threshold': 0.12132804095745087} [TRAIN] F4 suite28_context_only epoch=2/2 loss=1.57035 steps=345 sec=32.4 gradients={'context': 0.07589291781187057, 'course_meta': 0.08480383455753326, 'course_id': 0.01604081690311432, 'retention': 0.010651864111423492, 'grade_threshold': 0.0594293512403965} [PREDICTION] F4 suite28_context_only macro AP=0.22898 AUC=0.85641 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F4 suite28_mirt epoch=1/1 loss=1.67426 steps=345 sec=67.8 gradients={'context': 0.038263171911239624, 'course_meta': 0.03415913134813309, 'course_id': 0.020984139293432236, 'retention': 0.0008398705394938588, 'grade_threshold': 0.11414490640163422} [PREDICTION] F4 suite28_mirt macro AP=0.31920 AUC=0.88352 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F4 suite28_drrm epoch=1/1 loss=1.67645 steps=345 sec=56.7 gradients={'context': 0.03202936053276062, 'course_meta': 0.023032700642943382, 'course_id': 0.011789629235863686, 'retention': 0.0005364954704418778, 'grade_threshold': 0.11463884264230728} [GRADE LIKELIHOOD] observed_grade_rows= 41376 NLL= 1.6184308528900146 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F4 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 5072 244 15.260449 0.263722 2.214351 1.368348 0.799718 khoa_quan_tri_kinh_doanh_va_du_lich 6440 241 12.828416 0.326419 2.358317 1.571111 0.798874 khoa_quoc_te_hoc 2202 35 14.110354 0.254836 2.142306 1.425213 0.795748 khoa_tieng_anh 6636 68 16.922544 0.182310 1.849848 0.969213 0.779979 khoa_tieng_bo_dao_nha 869 64 18.379747 0.356824 2.465573 1.760832 0.800919 khoa_tieng_duc 1746 44 19.370562 0.241490 2.046853 1.253894 0.797570 khoa_tieng_han_quoc 3351 38 17.852283 0.157757 1.670526 1.049295 0.783706 khoa_tieng_italia 1795 34 20.255711 0.293751 2.301939 1.590607 0.801665 khoa_tieng_nga 1659 114 17.426764 0.364118 2.403800 1.716629 0.800253 khoa_tieng_nhat 2413 38 17.053045 0.171598 1.749863 1.009060 0.799876 khoa_tieng_phap 2275 19 17.197363 0.204875 2.006972 1.217208 0.801460 khoa_tieng_tay_ban_nha 1306 39 17.486217 0.255593 2.159705 1.485327 0.801600 khoa_tieng_trung_quoc 5107 104 17.037205 0.159841 1.676582 0.894253 0.775205 khoa_viet_nam_hoc 505 18 12.211882 0.367002 2.394844 1.637313 0.801841 [PREDICTION] F4 suite28_drrm macro AP=0.31163 AUC=0.87934 F4 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 4 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.957400 428.521393 -2.773940 0.295070 0.212030 1.044413 0.015541 5.268030e-14 0.0 0.0 4 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.970800 807.061829 -2.760177 0.301014 0.230181 1.059340 0.011546 3.972434e-14 0.0 0.0 4 khoa_quoc_te_hoc bottleneck 2202 course_requirement 0.915077 297.023163 -2.869634 0.458745 0.211617 1.019770 0.062082 3.822172e-14 0.0 0.0 4 khoa_tieng_anh bottleneck 5000 course_requirement 0.876200 506.444794 -3.027004 0.336416 0.193660 1.091898 0.023040 5.282046e-14 0.0 0.0 4 khoa_tieng_bo_dao_nha bottleneck 869 course_requirement 0.918297 97.617950 -2.952182 0.350281 0.191719 1.116408 0.022472 4.650891e-14 0.0 0.0 4 khoa_tieng_duc bottleneck 1746 course_requirement 0.938144 218.788086 -2.904971 0.309716 0.210580 1.128735 0.017725 5.389914e-14 0.0 0.0 4 khoa_tieng_han_quoc bottleneck 3351 course_requirement 0.910773 273.545807 -2.901225 0.304284 0.200897 1.152766 0.018556 3.676584e-14 0.0 0.0 4 khoa_tieng_italia bottleneck 1795 course_requirement 0.812813 123.469078 -2.920336 0.293249 0.174393 1.176213 0.019184 5.205699e-14 0.0 0.0 4 khoa_tieng_nga bottleneck 1659 course_requirement 0.788427 105.176613 -2.751580 0.263091 0.215635 1.104621 0.011254 4.656137e-14 0.0 0.0 4 khoa_tieng_nhat bottleneck 2413 course_requirement 0.926233 326.916290 -2.925291 0.287717 0.203296 1.227538 0.011690 5.201189e-14 0.0 0.0 4 khoa_tieng_phap bottleneck 2275 course_requirement 0.822418 199.740662 -3.086363 0.380559 0.168579 1.025086 0.046029 6.844869e-14 0.0 0.0 4 khoa_tieng_tay_ban_nha bottleneck 1306 course_requirement 0.873660 162.928787 -3.001728 0.261328 0.162521 1.148666 0.014489 5.563607e-14 0.0 0.0 4 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.798200 377.995605 -3.015709 0.311251 0.179562 1.132075 0.019105 4.093878e-14 0.0 0.0 4 khoa_viet_nam_hoc bottleneck 505 course_requirement 0.853465 140.714859 -2.997651 0.277295 0.171354 1.165630 0.013897 2.830301e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 4, 'rho_base': 0.8018413782119751, 'process_noise_diag': [0.01787467859685421, 0.018543867394328117, 0.018002664670348167, 0.018648086115717888], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.9090359807014465, 0.9156231880187988, 0.929514467716217, 0.9488476514816284], [0.9209206104278564, 0.9367674589157104, 0.9361827969551086, 0.9316805601119995], [0.9099164009094238, 0.9266641139984131, 0.94217848777771, 0.9414575099945068], [0.8017081022262573, 0.8862441182136536, 0.960546612739563, 0.9682252407073975], [0.912653923034668, 0.9206043481826782, 0.9320590496063232, 0.9276208281517029], [0.892439603805542, 0.9076219797134399, 0.9285205602645874, 0.9442266225814819], [0.8725488185882568, 0.9135769605636597, 0.9494593143463135, 0.9646821022033691], [0.9165383577346802, 0.9359011054039001, 0.9479001760482788, 0.9398351907730103], [0.9233174324035645, 0.9316678643226624, 0.9287784099578857, 0.9332073926925659], [0.8671514987945557, 0.9063283205032349, 0.9437845945358276, 0.9529328346252441], [0.8724356889724731, 0.9088793992996216, 0.9440410733222961, 0.9517427682876587], [0.8817846179008484, 0.9180684089660645, 0.9384104609489441, 0.9379295110702515], [0.837664008140564, 0.8955658674240112, 0.9499683976173401, 0.9615646600723267], [0.8640753626823425, 0.8927338719367981, 0.9239811897277832, 0.9381362199783325]]}] [INTERVENTION] F4 no_history mean_delta_ap=-0.08690 [INTERVENTION] F4 no_grade mean_delta_ap=+0.02160 [INTERVENTION] F4 mean_only mean_delta_ap=+0.00418 [INTERVENTION] F4 no_course_id mean_delta_ap=+0.00039 [INTERVENTION] F4 shuffle_historical_outcomes mean_delta_ap=-0.11929 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F5 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F5 coarse student-course-semester duplicate rows=0 / 419467 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F5 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F5 train_rows=419,467 target_rows=50,698 contexts=24 common_raw_full=136 FOLD 5 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 5 khoa_cong_nghe_thong_tin 6398 351 6047 0.054861 0.702094 0.940919 True 5 khoa_quan_tri_kinh_doanh_va_du_lich 7604 407 7197 0.053524 0.684245 0.976328 True 5 khoa_quoc_te_hoc 2674 58 2616 0.021690 0.620045 0.923336 True 5 khoa_tieng_anh 8345 132 8213 0.015818 0.840144 0.770042 True 5 khoa_tieng_bo_dao_nha 1138 80 1058 0.070299 0.755712 0.915641 True 5 khoa_tieng_duc 1975 59 1916 0.029873 0.652152 0.879494 True 5 khoa_tieng_han_quoc 3573 67 3506 0.018752 0.728519 0.934229 True 5 khoa_tieng_italia 2129 83 2046 0.038985 0.764678 0.818694 True 5 khoa_tieng_nga 2047 134 1913 0.065462 0.718124 0.891549 True 5 khoa_tieng_nhat 2593 69 2524 0.026610 0.711145 0.810644 True 5 khoa_tieng_phap 2983 57 2926 0.019108 0.709018 0.908146 True 5 khoa_tieng_tay_ban_nha 1606 84 1522 0.052304 0.684932 0.843088 True 5 khoa_tieng_trung_quoc 6995 155 6840 0.022159 0.815868 0.732809 True 5 khoa_viet_nam_hoc 638 61 577 0.095611 0.457680 0.877743 True [FULL FEATURE CONTROL] F5 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.29991 secs=204.6 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F5 suite28_context_only epoch=1/2 loss=1.68698 steps=349 sec=38.6 gradients={'context': 0.04692536219954491, 'course_meta': 0.02125496044754982, 'course_id': 0.013002390041947365, 'retention': 0.003296608105301857, 'grade_threshold': 0.11055701225996017} [TRAIN] F5 suite28_context_only epoch=2/2 loss=1.56809 steps=349 sec=43.0 gradients={'context': 0.2759491503238678, 'course_meta': 0.0709032341837883, 'course_id': 0.01640445739030838, 'retention': 0.014804193750023842, 'grade_threshold': 0.10682806372642517} [PREDICTION] F5 suite28_context_only macro AP=0.19285 AUC=0.79712 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F5 suite28_mirt epoch=1/1 loss=1.67029 steps=349 sec=65.3 gradients={'context': 0.0390448197722435, 'course_meta': 0.02508760802447796, 'course_id': 0.02486259676516056, 'retention': 0.0020079368259757757, 'grade_threshold': 0.10433438420295715} [PREDICTION] F5 suite28_mirt macro AP=0.24570 AUC=0.83178 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F5 suite28_drrm epoch=1/1 loss=1.67356 steps=349 sec=69.7 gradients={'context': 0.03949588164687157, 'course_meta': 0.023956334218382835, 'course_id': 0.014120007865130901, 'retention': 0.004340397194027901, 'grade_threshold': 0.10609578341245651} [GRADE LIKELIHOOD] observed_grade_rows= 50698 NLL= 1.6475143432617188 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F5 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 6398 351 12.534855 0.275648 2.345108 1.375931 0.552997 khoa_quan_tri_kinh_doanh_va_du_lich 7604 407 11.045108 0.339500 2.470264 1.536533 0.544794 khoa_quoc_te_hoc 2674 58 9.859760 0.257340 2.275531 1.370457 0.488443 khoa_tieng_anh 8345 132 11.362972 0.179250 1.953294 0.908250 0.661824 khoa_tieng_bo_dao_nha 1138 80 13.367311 0.404182 2.650301 1.778054 0.595401 khoa_tieng_duc 1975 59 11.113924 0.258070 2.210504 1.261868 0.513295 khoa_tieng_han_quoc 3573 67 11.081444 0.160740 1.745035 0.928114 0.573807 khoa_tieng_italia 2129 83 14.193048 0.295131 2.394419 1.553658 0.602460 khoa_tieng_nga 2047 134 15.065950 0.373798 2.568956 1.699580 0.564644 khoa_tieng_nhat 2593 69 9.636329 0.210877 1.978027 1.087262 0.558115 khoa_tieng_phap 2983 57 11.999330 0.243067 2.234873 1.266864 0.558094 khoa_tieng_tay_ban_nha 1606 84 12.583437 0.269880 2.288428 1.465946 0.539714 khoa_tieng_trung_quoc 6995 155 13.347820 0.172120 1.831073 0.861097 0.643864 khoa_viet_nam_hoc 638 61 6.097178 0.578553 3.005717 1.758068 0.359990 [PREDICTION] F5 suite28_drrm macro AP=0.24529 AUC=0.82761 F5 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 5 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.940600 552.690186 -2.788197 0.308968 0.245084 1.051032 0.015581 4.791588e-14 0.0 0.0 5 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.976600 883.873230 -2.716028 0.310652 0.269971 1.041585 0.013700 4.845028e-14 0.0 0.0 5 khoa_quoc_te_hoc bottleneck 2674 course_requirement 0.922962 331.706818 -2.755854 0.288091 0.246661 1.035192 0.015668 4.517490e-14 0.0 0.0 5 khoa_tieng_anh bottleneck 5000 course_requirement 0.769600 519.535217 -3.082088 0.352091 0.197935 1.036770 0.043904 4.445556e-14 0.0 0.0 5 khoa_tieng_bo_dao_nha bottleneck 1138 course_requirement 0.911248 127.873459 -2.941212 0.324972 0.216411 1.098013 0.019884 2.937236e-14 0.0 0.0 5 khoa_tieng_duc bottleneck 1975 course_requirement 0.874430 208.294678 -2.814031 0.302752 0.264024 1.097707 0.017285 3.578017e-14 0.0 0.0 5 khoa_tieng_han_quoc bottleneck 3573 course_requirement 0.933949 286.388763 -2.865400 0.310082 0.253314 1.128394 0.013215 5.121079e-14 0.0 0.0 5 khoa_tieng_italia bottleneck 2129 course_requirement 0.817285 166.198685 -2.864281 0.287755 0.222278 1.075489 0.021039 3.838029e-14 0.0 0.0 5 khoa_tieng_nga bottleneck 2047 course_requirement 0.881778 147.298004 -2.786910 0.344923 0.246513 1.068938 0.026092 3.791708e-14 0.0 0.0 5 khoa_tieng_nhat bottleneck 2593 course_requirement 0.807944 336.185883 -2.900616 0.291140 0.241687 1.075030 0.020231 2.819183e-14 0.0 0.0 5 khoa_tieng_phap bottleneck 2983 course_requirement 0.901777 230.150848 -3.056668 0.408819 0.200706 1.022252 0.049600 5.739920e-14 0.0 0.0 5 khoa_tieng_tay_ban_nha bottleneck 1606 course_requirement 0.839352 146.334991 -2.983523 0.300290 0.217454 1.083819 0.024814 3.450411e-14 0.0 0.0 5 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.731800 367.412415 -2.939443 0.267561 0.218271 1.089181 0.017654 6.181998e-14 0.0 0.0 5 khoa_viet_nam_hoc bottleneck 638 course_requirement 0.871473 118.026649 -2.907324 0.251257 0.243971 1.047621 0.024717 2.042169e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 5, 'rho_base': 0.8023962378501892, 'process_noise_diag': [0.017713898792862892, 0.018449967727065086, 0.017842749133706093, 0.01851738803088665], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.9126002788543701, 0.9197043180465698, 0.93133145570755, 0.9495512843132019], [0.9229637980461121, 0.9395331740379333, 0.9448648691177368, 0.9375923275947571], [0.9202286005020142, 0.9401149749755859, 0.9587734341621399, 0.9607215523719788], [0.8044758439064026, 0.8853111267089844, 0.945361852645874, 0.9546236991882324], [0.9229730367660522, 0.9319654107093811, 0.9431900382041931, 0.9470499753952026], [0.9007139205932617, 0.9238616228103638, 0.9388474822044373, 0.9473012685775757], [0.8647075295448303, 0.9054030776023865, 0.958733081817627, 0.9655802845954895], [0.9059967398643494, 0.9264185428619385, 0.9331241846084595, 0.9378032684326172], [0.9263862371444702, 0.9350131154060364, 0.9279889464378357, 0.936095118522644], [0.8731635808944702, 0.9156176447868347, 0.9469664096832275, 0.9599612951278687], [0.8757773637771606, 0.917790949344635, 0.9505575299263, 0.9620798230171204], [0.8771195411682129, 0.9123660922050476, 0.9333396553993225, 0.9455264210700989], [0.8414264917373657, 0.8981375098228455, 0.9451138973236084, 0.9567627310752869], [0.8644925355911255, 0.8959953784942627, 0.9393551349639893, 0.9522488117218018]]}] [INTERVENTION] F5 no_history mean_delta_ap=-0.02795 [INTERVENTION] F5 no_grade mean_delta_ap=+0.00136 [INTERVENTION] F5 mean_only mean_delta_ap=-0.00304 [INTERVENTION] F5 no_course_id mean_delta_ap=+0.00582 [INTERVENTION] F5 shuffle_historical_outcomes mean_delta_ap=-0.05465 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F6 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F6 coarse student-course-semester duplicate rows=0 / 470165 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F6 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F6 train_rows=470,165 target_rows=43,359 contexts=24 common_raw_full=136 FOLD 6 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 6 khoa_cong_nghe_thong_tin 5651 563 5088 0.099628 1.000000 0.985312 True 6 khoa_quan_tri_kinh_doanh_va_du_lich 7442 593 6849 0.079683 0.998388 0.949879 True 6 khoa_quoc_te_hoc 2466 60 2406 0.024331 1.000000 0.908354 True 6 khoa_tieng_anh 6204 122 6082 0.019665 0.978079 0.881689 True 6 khoa_tieng_bo_dao_nha 892 73 819 0.081839 1.000000 0.837444 True 6 khoa_tieng_duc 1839 84 1755 0.045677 0.992387 0.855356 True 6 khoa_tieng_han_quoc 3277 123 3154 0.037534 0.998474 0.763503 True 6 khoa_tieng_italia 2026 94 1932 0.046397 0.995064 0.813919 True 6 khoa_tieng_nga 1842 205 1637 0.111292 0.997286 0.775244 True 6 khoa_tieng_nhat 2530 88 2442 0.034783 0.997233 0.758893 True 6 khoa_tieng_phap 2263 50 2213 0.022095 0.996465 0.853292 True 6 khoa_tieng_tay_ban_nha 1333 85 1248 0.063766 1.000000 0.822206 True 6 khoa_tieng_trung_quoc 4990 167 4823 0.033467 0.972745 0.630661 True 6 khoa_viet_nam_hoc 604 88 516 0.145695 0.985099 0.864238 True [FULL FEATURE CONTROL] F6 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.32235 secs=218.7 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F6 suite28_context_only epoch=1/2 loss=1.67605 steps=412 sec=51.1 gradients={'context': 0.9868422746658325, 'course_meta': 0.4754655361175537, 'course_id': 0.18456366658210754, 'retention': 0.0, 'grade_threshold': 0.0} [TRAIN] F6 suite28_context_only epoch=2/2 loss=1.55142 steps=412 sec=48.5 gradients={'context': 0.7621210217475891, 'course_meta': 0.35988250374794006, 'course_id': 0.08778800070285797, 'retention': 0.0, 'grade_threshold': 0.4509366452693939} [PREDICTION] F6 suite28_context_only macro AP=0.21584 AUC=0.78793 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F6 suite28_mirt epoch=1/1 loss=1.65930 steps=412 sec=77.6 gradients={'context': 1.4084190130233765, 'course_meta': 0.6262544989585876, 'course_id': 0.3520624339580536, 'retention': 0.0, 'grade_threshold': 0.0} [PREDICTION] F6 suite28_mirt macro AP=0.27416 AUC=0.82699 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F6 suite28_drrm epoch=1/1 loss=1.66268 steps=412 sec=71.9 gradients={'context': 1.0522719621658325, 'course_meta': 0.5572040677070618, 'course_id': 0.20366400480270386, 'retention': 0.0, 'grade_threshold': 0.0} [GRADE LIKELIHOOD] observed_grade_rows= 43359 NLL= 1.6081258058547974 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F6 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 5651 563 15.711555 0.291432 2.184956 1.253280 0.798881 khoa_quan_tri_kinh_doanh_va_du_lich 7442 593 13.941414 0.326560 2.246989 1.395513 0.796905 khoa_quoc_te_hoc 2466 60 14.682482 0.242143 2.053526 1.153695 0.798485 khoa_tieng_anh 6204 122 16.083172 0.176883 1.695246 0.873921 0.781365 khoa_tieng_bo_dao_nha 892 73 19.080717 0.393554 2.378961 1.600361 0.799041 khoa_tieng_duc 1839 84 16.915171 0.268487 2.045982 1.176676 0.792592 khoa_tieng_han_quoc 3277 123 15.828807 0.162644 1.641598 1.020449 0.797262 khoa_tieng_italia 2026 94 17.839092 0.325250 2.237092 1.488659 0.794826 khoa_tieng_nga 1842 205 17.195982 0.378967 2.344412 1.578322 0.796691 khoa_tieng_nhat 2530 88 14.800395 0.201010 1.788201 1.102949 0.796144 khoa_tieng_phap 2263 50 15.707910 0.241834 2.028551 1.155278 0.796126 khoa_tieng_tay_ban_nha 1333 85 16.003000 0.274878 2.072159 1.364237 0.799083 khoa_tieng_trung_quoc 4990 167 16.274349 0.179488 1.705022 0.955959 0.777280 khoa_viet_nam_hoc 604 88 7.894040 0.690258 2.995956 1.901493 0.787025 [PREDICTION] F6 suite28_drrm macro AP=0.27014 AUC=0.82168 F6 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 6 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.984600 550.456604 -2.720170 0.428007 0.332818 1.063763 0.030970 3.845546e-14 0.0 0.0 6 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.946200 999.778198 -2.711891 0.399828 0.334022 1.077642 0.028794 5.638255e-14 0.0 0.0 6 khoa_quoc_te_hoc bottleneck 2466 course_requirement 0.907137 323.841431 -2.855928 0.502563 0.293340 1.034378 0.079259 4.103279e-14 0.0 0.0 6 khoa_tieng_anh bottleneck 5000 course_requirement 0.874400 574.865417 -2.991758 0.374620 0.272673 1.117334 0.040325 5.031657e-14 0.0 0.0 6 khoa_tieng_bo_dao_nha bottleneck 892 course_requirement 0.830717 106.656952 -2.952419 0.452073 0.271844 1.126390 0.059469 5.233809e-14 0.0 0.0 6 khoa_tieng_duc bottleneck 1839 course_requirement 0.851006 247.321915 -2.861943 0.422844 0.306202 1.132311 0.054077 2.998360e-14 0.0 0.0 6 khoa_tieng_han_quoc bottleneck 3277 course_requirement 0.762893 286.882507 -2.880848 0.400114 0.302739 1.183100 0.041997 3.338386e-14 0.0 0.0 6 khoa_tieng_italia bottleneck 2026 course_requirement 0.809970 148.068604 -2.855574 0.396078 0.288023 1.185876 0.043617 3.188621e-14 0.0 0.0 6 khoa_tieng_nga bottleneck 1842 course_requirement 0.768187 126.110748 -2.667623 0.393679 0.318036 1.125412 0.032326 3.154551e-14 0.0 0.0 6 khoa_tieng_nhat bottleneck 2530 course_requirement 0.754941 336.747833 -2.903635 0.395163 0.301879 1.176657 0.043486 2.887083e-14 0.0 0.0 6 khoa_tieng_phap bottleneck 2263 course_requirement 0.843129 237.368530 -3.137126 0.430564 0.224399 1.067049 0.059894 3.765087e-14 0.0 0.0 6 khoa_tieng_tay_ban_nha bottleneck 1333 course_requirement 0.821455 178.765945 -2.931311 0.375778 0.269860 1.152825 0.037632 4.363676e-14 0.0 0.0 6 khoa_tieng_trung_quoc bottleneck 4990 course_requirement 0.629058 314.868134 -2.939752 0.368690 0.251830 1.200279 0.036317 3.003989e-14 0.0 0.0 6 khoa_viet_nam_hoc bottleneck 604 course_requirement 0.832781 60.428806 -2.948434 0.368168 0.271549 1.067174 0.069515 3.641196e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 6, 'rho_base': 0.799168586730957, 'process_noise_diag': [0.017737112939357758, 0.018455151468515396, 0.017665714025497437, 0.01854531280696392], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.927361786365509, 0.9359413385391235, 0.9478776454925537, 0.975367546081543], [0.9294852614402771, 0.9496577978134155, 0.962547779083252, 0.9698278307914734], [0.9302760362625122, 0.9558185935020447, 0.9782582521438599, 0.9788745045661926], [0.8048297166824341, 0.8896823525428772, 0.9731713533401489, 0.9887901544570923], [0.9346835613250732, 0.9426577091217041, 0.9434255361557007, 0.9554926156997681], [0.9139545559883118, 0.9472230076789856, 0.9681916236877441, 0.9793024063110352], [0.8845598697662354, 0.9195818901062012, 0.9712273478507996, 0.9885146617889404], [0.9340993165969849, 0.9524340629577637, 0.961674690246582, 0.9568811058998108], [0.9434423446655273, 0.9493749737739563, 0.9508990049362183, 0.9597910642623901], [0.8962888121604919, 0.9457135796546936, 0.9766066670417786, 0.9913049936294556], [0.8964710235595703, 0.9427469372749329, 0.9679936170578003, 0.9926303625106812], [0.9158148169517517, 0.9456378221511841, 0.9588087797164917, 0.9730944633483887], [0.8654968738555908, 0.9265108704566956, 0.9862352013587952, 0.9963582754135132], [0.8882138729095459, 0.9138163328170776, 0.9656258821487427, 0.9729053974151611]]}] [INTERVENTION] F6 no_history mean_delta_ap=-0.04520 [INTERVENTION] F6 no_grade mean_delta_ap=+0.00243 [INTERVENTION] F6 mean_only mean_delta_ap=+0.00065 [INTERVENTION] F6 no_course_id mean_delta_ap=+0.00394 [INTERVENTION] F6 shuffle_historical_outcomes mean_delta_ap=-0.07135 ============================================================================================================================================ [PRODUCTION HISTORICAL EVALUATION] F7 ============================================================================================================================================ [EVENT IDENTITY AUDIT] F7 coarse student-course-semester duplicate rows=0 / 514858 (may represent separate registrations; do not silently deduplicate) [GRADE SUPPORT] F7 training numeric-grade coverage=1.00000; test-grade coverage (EVALUATION ONLY)=1.00000 [INPUT] F7 train_rows=514,858 target_rows=53,645 contexts=24 common_raw_full=136 FOLD 7 SUPPORT / DEVELOPMENT ELIGIBILITY -------------------------------------------------------------------------------- fold faculty rows positive negative prevalence seen_student_fraction known_course_fraction dev_support 7 khoa_cong_nghe_thong_tin 6926 1048 5878 0.151314 0.690297 0.887525 True 7 khoa_quan_tri_kinh_doanh_va_du_lich 8989 1123 7866 0.124930 0.654800 0.939704 True 7 khoa_quoc_te_hoc 2755 128 2627 0.046461 0.709982 0.888566 True 7 khoa_tieng_anh 8369 250 8119 0.029872 0.761859 0.869757 True 7 khoa_tieng_bo_dao_nha 1209 143 1066 0.118280 0.697270 0.785773 True 7 khoa_tieng_duc 2284 111 2173 0.048599 0.712785 0.934764 True 7 khoa_tieng_han_quoc 3434 125 3309 0.036401 0.760920 0.744030 True 7 khoa_tieng_italia 2453 254 2199 0.103547 0.783938 0.881370 True 7 khoa_tieng_nga 1924 154 1770 0.080042 0.649688 0.732328 True 7 khoa_tieng_nhat 2795 145 2650 0.051878 0.747406 0.730233 True 7 khoa_tieng_phap 2915 119 2796 0.040823 0.718696 0.845283 True 7 khoa_tieng_tay_ban_nha 1957 190 1767 0.097087 0.692386 0.820644 True 7 khoa_tieng_trung_quoc 6657 217 6440 0.032597 0.800361 0.701367 True 7 khoa_viet_nam_hoc 978 172 806 0.175869 0.677914 0.812883 True [FULL FEATURE CONTROL] F7 HGB uses 136 common raw +12 static+5 rebuilt as-of features+faculty onehot; macroAP=0.36458 secs=216.4 [PRODUCTION VARIANT] suite28_context_only epochs=2 [TRAIN] F7 suite28_context_only epoch=1/2 loss=1.67446 steps=415 sec=53.7 gradients={'context': 0.04677533358335495, 'course_meta': 0.026858912780880928, 'course_id': 0.01150491926819086, 'retention': 0.007521269842982292, 'grade_threshold': 0.10884508490562439} [TRAIN] F7 suite28_context_only epoch=2/2 loss=1.54938 steps=415 sec=44.8 gradients={'context': 0.04620722681283951, 'course_meta': 0.06248941645026207, 'course_id': 0.020049063488841057, 'retention': 0.006001679692417383, 'grade_threshold': 0.13133327662944794} [PREDICTION] F7 suite28_context_only macro AP=0.27687 AUC=0.80852 [PRODUCTION VARIANT] suite28_mirt epochs=1 [TRAIN] F7 suite28_mirt epoch=1/1 loss=1.65696 steps=415 sec=87.3 gradients={'context': 0.025735612958669662, 'course_meta': 0.02679736725986004, 'course_id': 0.02132047526538372, 'retention': 0.00619556475430727, 'grade_threshold': 0.10230959951877594} [PREDICTION] F7 suite28_mirt macro AP=0.30954 AUC=0.82625 [PRODUCTION VARIANT] suite28_drrm epochs=1 [TRAIN] F7 suite28_drrm epoch=1/1 loss=1.66078 steps=415 sec=70.4 gradients={'context': 0.028482845053076744, 'course_meta': 0.02810102142393589, 'course_id': 0.01274163369089365, 'retention': 0.011074144393205643, 'grade_threshold': 0.10373587161302567} [GRADE LIKELIHOOD] observed_grade_rows= 53645 NLL= 1.6217275857925415 grade_present_rate= 1.0 [GRADE-BIN COHERENCE] all six bins sum to 1; first bin equals failure marginal, PASS F7 UNCERTAINTY / SUPPORT BY FACULTY -------------------------------------------------------------------------------- faculty rows failures history_events projected_variance cov_trace readiness_norm retention khoa_cong_nghe_thong_tin 6926 1048 12.555443 0.397130 2.409268 1.315934 0.541683 khoa_quan_tri_kinh_doanh_va_du_lich 8989 1123 11.038047 0.494771 2.539485 1.442925 0.516223 khoa_quoc_te_hoc 2755 128 11.537568 0.299023 2.177888 1.111118 0.554605 khoa_tieng_anh 8369 250 10.138129 0.231201 1.909155 0.874662 0.595331 khoa_tieng_bo_dao_nha 1209 143 10.260546 0.520100 2.661229 1.813229 0.544780 khoa_tieng_duc 2284 111 10.972417 0.307309 2.137592 1.101610 0.557004 khoa_tieng_han_quoc 3434 125 10.700350 0.233613 1.863041 0.933691 0.594517 khoa_tieng_italia 2453 254 13.082756 0.408741 2.394667 1.488806 0.612093 khoa_tieng_nga 1924 154 12.083680 0.496589 2.588975 1.635644 0.506463 khoa_tieng_nhat 2795 145 8.078354 0.379244 2.263908 1.158031 0.580681 khoa_tieng_phap 2915 119 11.603087 0.332267 2.224377 1.205411 0.561426 khoa_tieng_tay_ban_nha 1957 190 11.443536 0.499008 2.551142 1.530727 0.540935 khoa_tieng_trung_quoc 6657 217 12.733814 0.251941 1.956738 0.922135 0.625956 khoa_viet_nam_hoc 978 172 4.747444 1.034134 3.466871 2.074901 0.529789 [PREDICTION] F7 suite28_drrm macro AP=0.30251 AUC=0.82196 F7 LEARNED COURSE REQUIREMENT GEOMETRY -------------------------------------------------------------------------------- fold faculty variant rows_sampled requirement_kind known_course_rate prior_course_mean requirement_mean requirement_std within_course_dimension_std centered_effective_rank largest_rank_eigenvalue smallest_rank_eigenvalue course_difficulty_mean course_difficulty_sd 7 khoa_cong_nghe_thong_tin bottleneck 5000 course_requirement 0.888200 671.080017 -2.853766 0.390617 0.318139 1.074171 0.028798 4.292023e-14 0.0 0.0 7 khoa_quan_tri_kinh_doanh_va_du_lich bottleneck 5000 course_requirement 0.930200 1087.240845 -2.726346 0.410595 0.377622 1.099707 0.019472 6.915554e-14 0.0 0.0 7 khoa_quoc_te_hoc bottleneck 2755 course_requirement 0.885662 406.859161 -2.830843 0.366552 0.313294 1.104860 0.021982 4.138097e-14 0.0 0.0 7 khoa_tieng_anh bottleneck 5000 course_requirement 0.865200 578.263428 -3.088718 0.401117 0.297524 1.208594 0.039150 5.384108e-14 0.0 0.0 7 khoa_tieng_bo_dao_nha bottleneck 1209 course_requirement 0.766749 133.595535 -2.934461 0.362298 0.301336 1.251109 0.029563 4.591415e-14 0.0 0.0 7 khoa_tieng_duc bottleneck 2284 course_requirement 0.918564 267.978546 -2.858519 0.377878 0.336878 1.343310 0.027883 4.390960e-14 0.0 0.0 7 khoa_tieng_han_quoc bottleneck 3434 course_requirement 0.739953 300.267029 -2.891559 0.357633 0.303019 1.246019 0.039372 4.542754e-14 0.0 0.0 7 khoa_tieng_italia bottleneck 2453 course_requirement 0.876070 191.960449 -2.910569 0.375748 0.309714 1.252619 0.020429 2.948117e-14 0.0 0.0 7 khoa_tieng_nga bottleneck 1924 course_requirement 0.715696 172.219330 -2.822166 0.416655 0.323288 1.275874 0.040680 2.843019e-14 0.0 0.0 7 khoa_tieng_nhat bottleneck 2795 course_requirement 0.725224 283.444366 -2.934758 0.347055 0.291365 1.242924 0.025254 3.321705e-14 0.0 0.0 7 khoa_tieng_phap bottleneck 2915 course_requirement 0.835678 273.590057 -3.138693 0.447635 0.290070 1.191133 0.036163 4.078238e-14 0.0 0.0 7 khoa_tieng_tay_ban_nha bottleneck 1957 course_requirement 0.816045 186.341339 -2.977696 0.366160 0.286928 1.227042 0.026434 3.041405e-14 0.0 0.0 7 khoa_tieng_trung_quoc bottleneck 5000 course_requirement 0.699000 375.764008 -2.937423 0.346907 0.306416 1.259248 0.022369 3.071563e-14 0.0 0.0 7 khoa_viet_nam_hoc bottleneck 978 course_requirement 0.812883 110.458076 -2.931630 0.349964 0.328458 1.112511 0.070277 7.548952e-14 0.0 0.0 [FILTER HEALTH] [{'fold': 7, 'rho_base': 0.795687735080719, 'process_noise_diag': [0.017676973715424538, 0.01842004433274269, 0.017605632543563843, 0.01828785240650177], 'damping': 0.65, 'discount': 0.7, 'tau': 0.65, 'ordinal_increment_by_faculty': [[0.9307863116264343, 0.9311320781707764, 0.9563056230545044, 0.9763439893722534], [0.9464136958122253, 0.9650344848632812, 0.9824622273445129, 0.9935541152954102], [0.937114953994751, 0.9667075872421265, 0.9894516468048096, 0.9925479888916016], [0.8052310943603516, 0.9011673927307129, 0.9802443385124207, 0.9871401786804199], [0.9362815618515015, 0.9458138942718506, 0.9619275331497192, 0.9624598026275635], [0.9064682722091675, 0.9425427913665771, 0.9750102758407593, 0.987633228302002], [0.8944982290267944, 0.9314225316047668, 0.9808540940284729, 0.9935753345489502], [0.9518203735351562, 0.9721384048461914, 0.9757789969444275, 0.9679394960403442], [0.9488409161567688, 0.9577898979187012, 0.9647682905197144, 0.9573147296905518], [0.8970072865486145, 0.9384444355964661, 0.9728880524635315, 0.982735276222229], [0.9007490873336792, 0.9518011808395386, 0.979886531829834, 0.9783377647399902], [0.8913699388504028, 0.9326057434082031, 0.9571876525878906, 0.9665040373802185], [0.8779064416885376, 0.9252880215644836, 0.9788264036178589, 0.9943656921386719], [0.8840597867965698, 0.9014502763748169, 0.9380379319190979, 0.961105227470398]]}] [INTERVENTION] F7 no_history mean_delta_ap=-0.00822 [INTERVENTION] F7 no_grade mean_delta_ap=-0.00678 [INTERVENTION] F7 mean_only mean_delta_ap=-0.00146 [INTERVENTION] F7 no_course_id mean_delta_ap=-0.00554 [INTERVENTION] F7 shuffle_historical_outcomes mean_delta_ap=-0.03192 F1–F7 MACRO PERFORMANCE -------------------------------------------------------------------------------- model ap auc brier ece10 fbeta_at_20 precision_at_20 recall_at_20 faculties cells council 0.383294 0.855707 0.051318 0.038571 0.456062 0.219794 0.713506 1 7 suite22 0.372436 0.883711 0.031744 0.016681 0.380613 0.155678 0.777482 14 98 suite25 0.369201 0.879386 0.031713 0.018808 0.375922 0.154109 0.766073 14 98 suite17 0.365412 0.881649 0.032409 0.021339 0.378685 0.154873 0.775350 14 98 suite23 0.365170 0.880518 0.031951 0.018668 0.377867 0.154685 0.771253 14 98 suite18 0.364397 0.881092 0.032422 0.021219 0.379101 0.155208 0.774247 14 98 suite21 0.361970 0.875832 0.032425 0.021011 0.373167 0.152872 0.761517 14 98 suite14 0.361533 0.870571 0.040871 0.047178 0.368112 0.151436 0.748357 14 98 committee 0.353855 0.870543 0.032294 0.019349 0.370098 0.151948 0.754042 14 98 suite24 0.352417 0.873192 0.032387 0.021546 0.369870 0.151991 0.751953 14 98 v2 0.351818 0.856084 0.099018 0.143562 0.449853 0.215630 0.707648 1 7 suite28_fulltab_hgb 0.327233 0.878434 0.032835 0.017741 0.374807 0.153604 0.763714 14 98 suite26 0.291830 0.855852 0.034405 0.020614 0.344558 0.141865 0.693187 14 98 suite28_mirt 0.273366 0.839618 0.045438 0.101409 0.338545 0.138885 0.690624 14 98 suite28_drrm 0.263163 0.834820 0.044921 0.098655 0.333114 0.136544 0.682102 14 98 suite28_context_only 0.208458 0.810133 0.043799 0.082626 0.307297 0.125848 0.633420 14 98 F6/F7 HISTORICAL HOLDOUT MACRO PERFORMANCE -------------------------------------------------------------------------------- model ap auc brier ece10 fbeta_at_20 precision_at_20 recall_at_20 faculties cells council 0.444927 0.833026 0.087385 0.015389 0.544238 0.381290 0.617044 1 2 v2 0.422654 0.826598 0.142849 0.162288 0.527904 0.370122 0.598342 1 2 suite25 0.369526 0.854274 0.052186 0.022578 0.454289 0.234216 0.689122 14 28 suite22 0.369159 0.855570 0.052508 0.023325 0.457353 0.235332 0.693883 14 28 suite14 0.360232 0.850384 0.068662 0.073762 0.452347 0.233303 0.686579 14 28 suite23 0.358265 0.850081 0.052685 0.022269 0.452802 0.233652 0.685744 14 28 suite17 0.357815 0.852825 0.053247 0.025095 0.453504 0.234047 0.687406 14 28 suite18 0.357805 0.852900 0.053261 0.025283 0.455319 0.235224 0.689275 14 28 suite21 0.357066 0.846907 0.053102 0.023850 0.446429 0.230796 0.676295 14 28 suite24 0.354254 0.846098 0.052775 0.024209 0.448556 0.232192 0.678738 14 28 committee 0.350152 0.843399 0.053294 0.024045 0.446492 0.231095 0.675598 14 28 suite28_fulltab_hgb 0.343467 0.849119 0.053533 0.027564 0.453934 0.234337 0.687078 14 28 suite26 0.302149 0.830961 0.056116 0.026845 0.418191 0.215635 0.636123 14 28 suite28_mirt 0.291853 0.826619 0.063078 0.089000 0.412954 0.212493 0.626521 14 28 suite28_drrm 0.286328 0.821818 0.062969 0.087333 0.409043 0.210182 0.621493 14 28 suite28_context_only 0.246353 0.798229 0.060916 0.060293 0.376850 0.193429 0.575860 14 28 F6/F7 ALL FACULTY AP -------------------------------------------------------------------------------- faculty committee council suite14 suite17 suite18 suite21 suite22 suite23 suite24 suite25 suite26 suite28_context_only suite28_drrm suite28_fulltab_hgb suite28_mirt v2 khoa_cong_nghe_thong_tin 0.412578 0.444927 0.422652 0.417591 0.414407 0.419185 0.424266 0.436409 0.422765 0.420793 0.353780 0.241118 0.287165 0.376854 0.301759 0.422654 khoa_quan_tri_kinh_doanh_va_du_lich 0.399207 NaN 0.401753 0.407873 0.408235 0.395783 0.429798 0.419884 0.412122 0.419906 0.388979 0.257885 0.336673 0.377625 0.347628 NaN khoa_quoc_te_hoc 0.261211 NaN 0.265888 0.267580 0.272936 0.261558 0.273008 0.275410 0.260185 0.272923 0.175411 0.162080 0.201535 0.222562 0.210079 NaN khoa_tieng_anh 0.254547 NaN 0.250760 0.262702 0.265981 0.258094 0.260863 0.262926 0.266065 0.257801 0.180283 0.131227 0.169567 0.196693 0.168145 NaN khoa_tieng_bo_dao_nha 0.464251 NaN 0.451954 0.468486 0.466295 0.470990 0.462717 0.448702 0.458294 0.458830 0.386121 0.340105 0.380051 0.431298 0.379766 NaN khoa_tieng_duc 0.375697 NaN 0.391938 0.388699 0.391861 0.385958 0.402625 0.362176 0.365631 0.400027 0.289314 0.272236 0.317922 0.344088 0.327757 NaN khoa_tieng_han_quoc 0.315879 NaN 0.332499 0.333005 0.332781 0.336184 0.358607 0.326835 0.345569 0.360616 0.260435 0.189340 0.211152 0.365792 0.224615 NaN khoa_tieng_italia 0.302636 NaN 0.304270 0.308005 0.306753 0.305668 0.314573 0.310503 0.304122 0.316856 0.270305 0.200804 0.243404 0.315728 0.226326 NaN khoa_tieng_nga 0.377031 NaN 0.383116 0.384702 0.385985 0.377869 0.393063 0.377247 0.370832 0.390821 0.351734 0.302841 0.350403 0.388311 0.354224 NaN khoa_tieng_nhat 0.197887 NaN 0.212153 0.200121 0.199897 0.211685 0.219105 0.211387 0.217737 0.230154 0.174530 0.144763 0.161051 0.199434 0.174216 NaN khoa_tieng_phap 0.276935 NaN 0.291062 0.283644 0.283197 0.276015 0.307482 0.301838 0.265082 0.317518 0.227000 0.210482 0.233100 0.304818 0.225979 NaN khoa_tieng_tay_ban_nha 0.387535 NaN 0.413933 0.395079 0.393249 0.408967 0.422826 0.396799 0.390748 0.428023 0.354933 0.288235 0.336541 0.397836 0.357868 NaN khoa_tieng_trung_quoc 0.249532 NaN 0.275808 0.260093 0.259264 0.261752 0.276961 0.258389 0.255801 0.275364 0.169586 0.147405 0.190658 0.225559 0.189682 NaN khoa_viet_nam_hoc 0.627200 NaN 0.645454 0.631835 0.628428 0.629217 0.622336 0.627200 0.624606 0.623726 0.647668 0.560426 0.589370 0.661942 0.597902 NaN ALIGNED PAIRED INCREMENTS -------------------------------------------------------------------------------- comparison cells mean_ap_delta wins mean_f2_delta drrm minus suite28_context_only 28 0.039975 28 0.032194 drrm minus suite28_mirt 28 -0.005525 9 -0.003911 drrm minus committee 28 -0.063824 4 -0.037449 drrm minus suite22 28 -0.082831 2 -0.048310 drrm minus suite25 28 -0.083197 3 -0.045246 drrm minus council 2 -0.157762 0 -0.131032 STUDENT-CLUSTER BOOTSTRAP (POOLED F6/F7 PER FACULTY) -------------------------------------------------------------------------------- faculty reference delta_ap ci_low ci_high valid_bootstraps khoa_cong_nghe_thong_tin committee -0.126884 -0.148767 -0.099313 200 khoa_cong_nghe_thong_tin suite22 -0.136479 -0.162166 -0.110604 200 khoa_cong_nghe_thong_tin suite25 -0.140014 -0.169452 -0.111372 200 khoa_cong_nghe_thong_tin suite28_mirt -0.015071 -0.019369 -0.010791 200 khoa_cong_nghe_thong_tin suite28_context_only 0.052299 0.040282 0.068263 200 khoa_cong_nghe_thong_tin council -0.159773 -0.185338 -0.131073 200 khoa_quan_tri_kinh_doanh_va_du_lich committee -0.060088 -0.083691 -0.039508 200 khoa_quan_tri_kinh_doanh_va_du_lich suite22 -0.089433 -0.112893 -0.066263 200 khoa_quan_tri_kinh_doanh_va_du_lich suite25 -0.079386 -0.104869 -0.060617 200 khoa_quan_tri_kinh_doanh_va_du_lich suite28_mirt -0.012340 -0.016344 -0.007448 200 khoa_quan_tri_kinh_doanh_va_du_lich suite28_context_only 0.076985 0.061155 0.092983 200 khoa_quoc_te_hoc committee -0.008789 -0.067625 0.073218 200 khoa_quoc_te_hoc suite22 -0.061251 -0.119528 -0.007299 200 khoa_quoc_te_hoc suite25 -0.050192 -0.107933 0.017556 200 khoa_quoc_te_hoc suite28_mirt -0.015950 -0.033199 -0.001166 200 khoa_quoc_te_hoc suite28_context_only 0.037201 0.002898 0.091933 200 khoa_tieng_anh committee -0.059961 -0.102802 -0.012119 200 khoa_tieng_anh suite22 -0.078030 -0.116305 -0.034341 200 khoa_tieng_anh suite25 -0.079416 -0.117716 -0.035579 200 khoa_tieng_anh suite28_mirt -0.000539 -0.009706 0.010245 200 khoa_tieng_anh suite28_context_only 0.038484 0.012497 0.069400 200 khoa_tieng_bo_dao_nha committee -0.074960 -0.124027 -0.014646 200 khoa_tieng_bo_dao_nha suite22 -0.076313 -0.125208 -0.016386 200 khoa_tieng_bo_dao_nha suite25 -0.080271 -0.130381 -0.011893 200 khoa_tieng_bo_dao_nha suite28_mirt -0.002219 -0.023843 0.016612 200 khoa_tieng_bo_dao_nha suite28_context_only 0.035388 0.007071 0.064568 200 khoa_tieng_duc committee -0.057208 -0.121426 0.011938 200 khoa_tieng_duc suite22 -0.085932 -0.148224 -0.014608 200 khoa_tieng_duc suite25 -0.086500 -0.145516 -0.019833 200 khoa_tieng_duc suite28_mirt -0.008231 -0.020336 0.004491 200 khoa_tieng_duc suite28_context_only 0.046347 0.009521 0.090981 200 khoa_tieng_han_quoc committee -0.084019 -0.160939 -0.011757 200 khoa_tieng_han_quoc suite22 -0.126663 -0.178954 -0.068235 200 khoa_tieng_han_quoc suite25 -0.122447 -0.178223 -0.054860 200 khoa_tieng_han_quoc suite28_mirt -0.011650 -0.020448 -0.000623 200 khoa_tieng_han_quoc suite28_context_only 0.020916 0.003216 0.043219 200 khoa_tieng_italia committee -0.011775 -0.067773 0.032271 200 khoa_tieng_italia suite22 -0.070001 -0.135698 -0.025493 200 khoa_tieng_italia suite25 -0.046687 -0.102920 -0.002510 200 khoa_tieng_italia suite28_mirt 0.016525 0.006245 0.027385 200 khoa_tieng_italia suite28_context_only 0.052328 0.033889 0.079149 200 khoa_tieng_nga committee -0.031054 -0.076904 0.021938 200 khoa_tieng_nga suite22 -0.055571 -0.101217 -0.009767 200 khoa_tieng_nga suite25 -0.021021 -0.057426 0.032466 200 khoa_tieng_nga suite28_mirt 0.005476 -0.008914 0.019504 200 khoa_tieng_nga suite28_context_only 0.047884 0.012955 0.086027 200 khoa_tieng_nhat committee -0.018149 -0.052964 0.030416 200 khoa_tieng_nhat suite22 -0.037949 -0.079421 -0.002418 200 khoa_tieng_nhat suite25 -0.045112 -0.081013 -0.003546 200 khoa_tieng_nhat suite28_mirt -0.012284 -0.021959 -0.000086 200 khoa_tieng_nhat suite28_context_only 0.019057 0.008695 0.033865 200 khoa_tieng_phap committee -0.024906 -0.107451 0.022480 200 khoa_tieng_phap suite22 -0.058819 -0.118603 -0.011032 200 khoa_tieng_phap suite25 -0.065579 -0.133729 -0.007790 200 khoa_tieng_phap suite28_mirt -0.001561 -0.015729 0.016193 200 khoa_tieng_phap suite28_context_only 0.024323 -0.007558 0.061699 200 khoa_tieng_tay_ban_nha committee -0.005088 -0.076304 0.070791 200 khoa_tieng_tay_ban_nha suite22 -0.054804 -0.118613 0.010437 200 khoa_tieng_tay_ban_nha suite25 -0.056507 -0.118426 0.014507 200 khoa_tieng_tay_ban_nha suite28_mirt -0.024907 -0.038830 -0.007190 200 khoa_tieng_tay_ban_nha suite28_context_only 0.048261 0.027040 0.074264 200 khoa_tieng_trung_quoc committee -0.050824 -0.095072 -0.006231 200 khoa_tieng_trung_quoc suite22 -0.083980 -0.122962 -0.041937 200 khoa_tieng_trung_quoc suite25 -0.078443 -0.123480 -0.032969 200 khoa_tieng_trung_quoc suite28_mirt 0.002396 -0.009052 0.011040 200 khoa_tieng_trung_quoc suite28_context_only 0.041481 0.025104 0.056385 200 khoa_viet_nam_hoc committee -0.056693 -0.093994 -0.002130 200 khoa_viet_nam_hoc suite22 -0.049122 -0.087678 0.003589 200 khoa_viet_nam_hoc suite25 -0.053470 -0.086114 -0.003230 200 khoa_viet_nam_hoc suite28_mirt -0.017233 -0.038146 0.012782 200 khoa_viet_nam_hoc suite28_context_only 0.030780 -0.007003 0.077120 200 FROZEN HISTORICAL COUNCIL OVERLAP (NOT ROW-ALIGNED) -------------------------------------------------------------------------------- fold faculty ap fbeta_at_20 council_historical_ap council_historical_fixed_f2 delta_ap 6 khoa_cong_nghe_thong_tin 0.235645 0.409521 0.4203 0.5617 -0.184655 6 khoa_quan_tri_kinh_doanh_va_du_lich 0.308964 0.503886 0.4261 0.4460 -0.117136 6 khoa_quoc_te_hoc 0.123194 0.245566 0.2698 0.2875 -0.146606 6 khoa_tieng_bo_dao_nha 0.384397 0.521277 0.5010 0.5306 -0.116603 6 khoa_tieng_duc 0.378019 0.497159 0.4457 0.5673 -0.067681 6 khoa_tieng_han_quoc 0.186483 0.322581 0.2531 0.3343 -0.066617 6 khoa_tieng_italia 0.150999 0.339309 0.2294 0.4056 -0.078401 6 khoa_tieng_nga 0.460537 0.509259 0.4951 0.4968 -0.034563 6 khoa_tieng_nhat 0.106470 0.244755 0.1763 0.3623 -0.069830 6 khoa_tieng_phap 0.216410 0.306279 0.2728 0.4936 -0.056390 6 khoa_tieng_tay_ban_nha 0.246757 0.428336 0.3751 0.5086 -0.128343 6 khoa_tieng_trung_quoc 0.178167 0.321128 0.3375 0.3759 -0.159333 6 khoa_viet_nam_hoc 0.665616 0.644820 0.6504 0.7203 0.015216 7 khoa_cong_nghe_thong_tin 0.338685 0.416891 0.4696 0.5749 -0.130915 7 khoa_quan_tri_kinh_doanh_va_du_lich 0.364383 0.480127 0.4335 0.4896 -0.069117 7 khoa_quoc_te_hoc 0.279875 0.428034 0.2446 0.0097 0.035275 7 khoa_tieng_bo_dao_nha 0.375705 0.442260 0.4225 0.5254 -0.046795 7 khoa_tieng_duc 0.257826 0.438402 0.3134 0.4072 -0.055574 7 khoa_tieng_han_quoc 0.235822 0.383319 0.3786 0.4971 -0.142778 7 khoa_tieng_italia 0.335809 0.484406 0.3767 0.4442 -0.040891 7 khoa_tieng_nga 0.240269 0.409590 0.2841 0.4284 -0.043831 7 khoa_tieng_nhat 0.215633 0.364355 0.2626 0.4286 -0.046967 7 khoa_tieng_phap 0.249790 0.410765 0.2830 0.3103 -0.033210 7 khoa_tieng_tay_ban_nha 0.426324 0.534318 0.4040 0.5851 0.022324 7 khoa_tieng_trung_quoc 0.203149 0.322874 0.1597 0.0666 0.043449 7 khoa_viet_nam_hoc 0.513124 0.509050 0.5844 0.6758 -0.071276 [COUNCIL CAUTION] F6/F7 archive has 13 faculty cells, English absent; archival values may be rounded; no cluster CI is possible from this CSV alone. F6/F7 FROZEN OPERATIONAL POLICY, FACULTY-MEAN -------------------------------------------------------------------------------- model policy precision recall fbeta flag_rate council fixed 0.330473 0.716895 0.579289 0.269036 v2 fixed 0.320999 0.702715 0.566065 0.271027 suite23 fixed 0.264194 0.607644 0.471903 0.163172 suite25 fixed 0.302361 0.576108 0.471566 0.151387 suite22 fixed 0.309207 0.574077 0.469158 0.144843 suite21 fixed 0.271450 0.609221 0.467293 0.171616 suite18 fixed 0.285552 0.597974 0.466852 0.162572 suite17 fixed 0.287958 0.594681 0.463354 0.163404 committee fixed 0.258570 0.605690 0.463230 0.172904 suite14 fixed 0.229138 0.686435 0.462320 0.230498 suite28_fulltab_hgb fixed 0.234912 0.657139 0.457765 0.206643 suite24 fixed 0.301330 0.536647 0.445405 0.139989 suite28_drrm fixed 0.216561 0.590936 0.425872 0.194249 suite28_mirt fixed 0.232489 0.549136 0.419350 0.172850 suite26 fixed 0.286960 0.462296 0.386016 0.135153 suite28_context_only fixed 0.206402 0.502755 0.379022 0.176729 v2 rate 0.428806 0.488678 0.471810 0.140978 council rate 0.458002 0.468214 0.462847 0.126021 suite28_fulltab_hgb rate 0.337773 0.455097 0.414163 0.095095 suite22 rate 0.380351 0.432390 0.411556 0.080839 suite14 rate 0.370636 0.448572 0.407314 0.091877 suite21 rate 0.362505 0.423536 0.400335 0.082063 suite23 rate 0.368739 0.414870 0.397998 0.075753 committee rate 0.362619 0.406830 0.390732 0.077998 suite17 rate 0.387220 0.398548 0.389282 0.072586 suite25 rate 0.397696 0.397191 0.386990 0.074934 suite18 rate 0.379096 0.399653 0.385249 0.076339 suite26 rate 0.322183 0.395993 0.365949 0.086807 suite28_drrm rate 0.287992 0.408688 0.364231 0.101114 suite28_mirt rate 0.310991 0.387305 0.357599 0.092932 suite24 rate 0.384601 0.362757 0.350899 0.071257 suite28_context_only rate 0.226629 0.417312 0.349065 0.133060 IT OPERATIONAL CONFUSION MATRICES -------------------------------------------------------------------------------- fold faculty model policy rows failures precision recall f1 fbeta tp fp fn tn flag_rate accuracy 6 khoa_cong_nghe_thong_tin committee fixed 5651 563 0.285817 0.705151 0.406762 0.545180 397 992 166 4096 0.245797 0.795081 6 khoa_cong_nghe_thong_tin committee rate 5651 563 0.368919 0.484902 0.419033 0.456217 273 467 290 4621 0.130950 0.866041 7 khoa_cong_nghe_thong_tin committee fixed 6926 1048 0.337953 0.721374 0.460274 0.587961 756 1481 292 4397 0.322986 0.744008 7 khoa_cong_nghe_thong_tin committee rate 6926 1048 0.502756 0.435115 0.466496 0.447146 456 451 592 5427 0.130956 0.849408 6 khoa_cong_nghe_thong_tin suite22 fixed 5651 563 0.291045 0.692718 0.409879 0.542873 390 950 173 4138 0.237126 0.801274 6 khoa_cong_nghe_thong_tin suite22 rate 5651 563 0.376687 0.545293 0.445573 0.500489 307 508 256 4580 0.144222 0.864803 7 khoa_cong_nghe_thong_tin suite22 fixed 6926 1048 0.331007 0.724237 0.454355 0.585197 759 1534 289 4344 0.331071 0.736789 7 khoa_cong_nghe_thong_tin suite22 rate 6926 1048 0.477477 0.455153 0.466048 0.459449 477 522 571 5356 0.144239 0.842189 6 khoa_cong_nghe_thong_tin suite25 fixed 5651 563 0.331337 0.589698 0.424281 0.510141 332 670 231 4418 0.177314 0.840559 6 khoa_cong_nghe_thong_tin suite25 rate 5651 563 0.369759 0.516874 0.431111 0.478776 291 496 272 4592 0.139267 0.864095 7 khoa_cong_nghe_thong_tin suite25 fixed 6926 1048 0.338373 0.726145 0.461632 0.590747 761 1488 287 4390 0.324718 0.743719 7 khoa_cong_nghe_thong_tin suite25 rate 6926 1048 0.480829 0.442748 0.461003 0.449874 464 501 584 5377 0.139330 0.843344 6 khoa_cong_nghe_thong_tin suite23 fixed 5651 563 0.326679 0.639432 0.432432 0.536673 360 742 203 4346 0.195010 0.832773 6 khoa_cong_nghe_thong_tin suite23 rate 5651 563 0.399711 0.492007 0.441083 0.470289 277 416 286 4672 0.122633 0.875774 7 khoa_cong_nghe_thong_tin suite23 fixed 6926 1048 0.361691 0.677481 0.471604 0.576767 710 1253 338 4625 0.283425 0.770286 7 khoa_cong_nghe_thong_tin suite23 rate 6926 1048 0.513545 0.416031 0.459673 0.432454 436 413 612 5465 0.122582 0.852007 6 khoa_cong_nghe_thong_tin suite24 fixed 5651 563 0.364190 0.531083 0.432081 0.486495 299 522 264 4566 0.145284 0.860910 6 khoa_cong_nghe_thong_tin suite24 rate 5651 563 0.377809 0.477798 0.421961 0.453779 269 443 294 4645 0.125995 0.869581 7 khoa_cong_nghe_thong_tin suite24 fixed 6926 1048 0.342011 0.727099 0.465201 0.593458 762 1466 286 4412 0.321686 0.747040 7 khoa_cong_nghe_thong_tin suite24 rate 6926 1048 0.499427 0.416031 0.453930 0.430405 436 437 612 5441 0.126047 0.848542 6 khoa_cong_nghe_thong_tin suite17 fixed 5651 563 0.298742 0.674956 0.414169 0.539160 380 892 183 4196 0.225093 0.809768 6 khoa_cong_nghe_thong_tin suite17 rate 5651 563 0.396440 0.435169 0.414903 0.426829 245 373 318 4715 0.109361 0.877721 7 khoa_cong_nghe_thong_tin suite17 fixed 6926 1048 0.359285 0.651718 0.463208 0.560479 683 1218 365 4660 0.274473 0.771441 7 khoa_cong_nghe_thong_tin suite17 rate 6926 1048 0.528402 0.381679 0.443213 0.404122 400 357 648 5521 0.109298 0.854895 6 khoa_cong_nghe_thong_tin suite18 fixed 5651 563 0.273256 0.751332 0.400758 0.556579 423 1125 140 3963 0.273934 0.776146 6 khoa_cong_nghe_thong_tin suite18 rate 5651 563 0.378653 0.529307 0.441481 0.490293 298 489 265 4599 0.139267 0.866572 7 khoa_cong_nghe_thong_tin suite18 fixed 6926 1048 0.331126 0.715649 0.452762 0.580765 750 1515 298 4363 0.327029 0.738233 7 khoa_cong_nghe_thong_tin suite18 rate 6926 1048 0.478756 0.440840 0.459016 0.447935 462 503 586 5375 0.139330 0.842766 6 khoa_cong_nghe_thong_tin suite21 fixed 5651 563 0.305556 0.664298 0.418579 0.537975 374 850 189 4238 0.216599 0.816139 6 khoa_cong_nghe_thong_tin suite21 rate 5651 563 0.385491 0.481350 0.428120 0.458545 271 432 292 4656 0.124403 0.871881 7 khoa_cong_nghe_thong_tin suite21 fixed 6926 1048 0.306911 0.750000 0.435578 0.581964 786 1775 262 4103 0.369766 0.705891 7 khoa_cong_nghe_thong_tin suite21 rate 6926 1048 0.511034 0.419847 0.460974 0.435385 440 421 608 5457 0.124314 0.851429 6 khoa_cong_nghe_thong_tin suite14 fixed 5651 563 0.214448 0.838366 0.341534 0.529980 472 1729 91 3359 0.389489 0.677933 6 khoa_cong_nghe_thong_tin suite14 rate 5651 563 0.329681 0.587922 0.422463 0.508292 331 673 232 4415 0.177668 0.839851 7 khoa_cong_nghe_thong_tin suite14 fixed 6926 1048 0.296684 0.811069 0.434449 0.602239 850 2015 198 3863 0.413659 0.680479 7 khoa_cong_nghe_thong_tin suite14 rate 6926 1048 0.439837 0.516221 0.474978 0.498893 541 689 507 5189 0.177592 0.827317 6 khoa_cong_nghe_thong_tin suite26 fixed 5651 563 0.394541 0.282416 0.329193 0.299435 159 244 404 4844 0.071315 0.885330 6 khoa_cong_nghe_thong_tin suite26 rate 5651 563 0.346939 0.392540 0.368333 0.382485 221 416 342 4672 0.112723 0.865864 7 khoa_cong_nghe_thong_tin suite26 fixed 6926 1048 0.312436 0.584924 0.407309 0.498050 613 1349 435 4529 0.283280 0.742420 7 khoa_cong_nghe_thong_tin suite26 rate 6926 1048 0.434615 0.323473 0.370897 0.340909 339 441 709 5437 0.112619 0.833959 6 khoa_cong_nghe_thong_tin council fixed 5651 563 0.295570 0.722913 0.419588 0.560761 407 970 156 4118 0.243674 0.800743 6 khoa_cong_nghe_thong_tin council rate 5651 563 0.401685 0.507993 0.448627 0.482456 286 426 277 4662 0.125995 0.875597 7 khoa_cong_nghe_thong_tin council fixed 6926 1048 0.365375 0.710878 0.482669 0.597817 745 1294 303 4584 0.294398 0.769420 7 khoa_cong_nghe_thong_tin council rate 6926 1048 0.514318 0.428435 0.467465 0.443238 449 424 599 5454 0.126047 0.852296 6 khoa_cong_nghe_thong_tin v2 fixed 5651 563 0.292630 0.719361 0.416025 0.556931 405 979 158 4109 0.244912 0.798797 6 khoa_cong_nghe_thong_tin v2 rate 5651 563 0.368883 0.522202 0.432353 0.482125 294 503 269 4585 0.141037 0.863387 7 khoa_cong_nghe_thong_tin v2 fixed 6926 1048 0.349368 0.686069 0.462975 0.575200 719 1339 329 4539 0.297141 0.759168 7 khoa_cong_nghe_thong_tin v2 rate 6926 1048 0.488730 0.455153 0.471344 0.461494 477 499 571 5379 0.140918 0.845510 6 khoa_cong_nghe_thong_tin suite28_fulltab_hgb fixed 5651 563 0.261429 0.650089 0.372899 0.501095 366 1034 197 4054 0.247744 0.782162 6 khoa_cong_nghe_thong_tin suite28_fulltab_hgb rate 5651 563 0.309386 0.474245 0.374474 0.428571 267 596 296 4492 0.152716 0.842152 7 khoa_cong_nghe_thong_tin suite28_fulltab_hgb fixed 6926 1048 0.338891 0.717557 0.460361 0.586492 752 1467 296 4411 0.320387 0.745452 7 khoa_cong_nghe_thong_tin suite28_fulltab_hgb rate 6926 1048 0.464522 0.468511 0.466508 0.467708 491 566 557 5312 0.152613 0.837857 6 khoa_cong_nghe_thong_tin suite28_context_only fixed 5651 563 0.170569 0.634103 0.268825 0.410817 357 1736 206 3352 0.370377 0.656344 6 khoa_cong_nghe_thong_tin suite28_context_only rate 5651 563 0.180365 0.420959 0.252531 0.332305 237 1077 326 4011 0.232525 0.751725 7 khoa_cong_nghe_thong_tin suite28_context_only fixed 6926 1048 0.292148 0.482824 0.364029 0.427076 506 1226 542 4652 0.250072 0.744730 7 khoa_cong_nghe_thong_tin suite28_context_only rate 6926 1048 0.296710 0.456107 0.359534 0.411856 478 1133 570 4745 0.232602 0.754115 6 khoa_cong_nghe_thong_tin suite28_mirt fixed 5651 563 0.203340 0.735346 0.318584 0.482743 414 1622 149 3466 0.360290 0.686604 6 khoa_cong_nghe_thong_tin suite28_mirt rate 5651 563 0.249141 0.515098 0.335843 0.424473 290 874 273 4214 0.205981 0.797027 7 khoa_cong_nghe_thong_tin suite28_mirt fixed 6926 1048 0.314788 0.631679 0.420184 0.525814 662 1441 386 4437 0.303638 0.736211 7 khoa_cong_nghe_thong_tin suite28_mirt rate 6926 1048 0.347125 0.472328 0.400162 0.440548 495 931 553 4947 0.205891 0.785735 6 khoa_cong_nghe_thong_tin suite28_drrm fixed 5651 563 0.186266 0.770870 0.300035 0.473592 434 1896 129 3192 0.412316 0.641656 6 khoa_cong_nghe_thong_tin suite28_drrm rate 5651 563 0.235991 0.531083 0.326776 0.424837 299 968 264 4120 0.224208 0.781985 7 khoa_cong_nghe_thong_tin suite28_drrm fixed 6926 1048 0.287433 0.661260 0.400694 0.524761 693 1718 355 4160 0.348109 0.700693 7 khoa_cong_nghe_thong_tin suite28_drrm rate 6926 1048 0.321958 0.477099 0.384468 0.435161 500 1053 548 4825 0.224228 0.768842 POSTFIT MECHANISM INTERVENTIONS BY HOLDOUT -------------------------------------------------------------------------------- intervention mean_delta_ap harmed_cells cells score_mae mean_only -0.000406 13 28 0.032564 no_course_id -0.000799 13 28 0.008320 no_grade -0.002176 16 28 0.015066 no_history -0.026709 22 28 0.023817 shuffle_historical_outcomes -0.051637 26 28 0.020104 HOLDOUT SUPPORT / COLD-START STRATA (EXCERPT) -------------------------------------------------------------------------------- fold faculty stratum group model rows failures ap auc 6 khoa_cong_nghe_thong_tin history 1-4 primary 512 110 0.252176 0.590886 6 khoa_cong_nghe_thong_tin history 1-4 committee 512 110 0.396101 0.723293 6 khoa_cong_nghe_thong_tin history 5-16 primary 2247 305 0.253692 0.707013 6 khoa_cong_nghe_thong_tin history 5-16 committee 2247 305 0.370291 0.784979 6 khoa_cong_nghe_thong_tin history 17+ primary 2892 148 0.233804 0.803941 6 khoa_cong_nghe_thong_tin history 17+ committee 2892 148 0.385965 0.875308 6 khoa_cong_nghe_thong_tin course_support unseen primary 83 1 0.014706 0.182927 6 khoa_cong_nghe_thong_tin course_support unseen committee 83 1 0.029412 0.597561 6 khoa_cong_nghe_thong_tin course_support rare1-9 primary 6 0 NaN NaN 6 khoa_cong_nghe_thong_tin course_support rare1-9 committee 6 0 NaN NaN 6 khoa_cong_nghe_thong_tin course_support supported10+ primary 5562 562 0.236186 0.759989 6 khoa_cong_nghe_thong_tin course_support supported10+ committee 5562 562 0.374693 0.836018 6 khoa_cong_nghe_thong_tin student_identity recorded_history primary 5651 563 0.235645 0.762060 6 khoa_cong_nghe_thong_tin student_identity recorded_history committee 5651 563 0.372426 0.836092 6 khoa_quan_tri_kinh_doanh_va_du_lich history 0 primary 12 5 0.549394 0.371429 6 khoa_quan_tri_kinh_doanh_va_du_lich history 0 committee 12 5 0.597778 0.714286 6 khoa_quan_tri_kinh_doanh_va_du_lich history 1-4 primary 802 173 0.285880 0.656400 6 khoa_quan_tri_kinh_doanh_va_du_lich history 1-4 committee 802 173 0.356261 0.684208 6 khoa_quan_tri_kinh_doanh_va_du_lich history 5-16 primary 3734 311 0.321947 0.826073 6 khoa_quan_tri_kinh_doanh_va_du_lich history 5-16 committee 3734 311 0.383756 0.829797 6 khoa_quan_tri_kinh_doanh_va_du_lich history 17+ primary 2894 104 0.384812 0.896528 6 khoa_quan_tri_kinh_doanh_va_du_lich history 17+ committee 2894 104 0.423076 0.900806 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support unseen primary 373 31 0.281375 0.817299 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support unseen committee 373 31 0.272844 0.837012 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support rare1-9 primary 157 0 NaN NaN 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support rare1-9 committee 157 0 NaN NaN 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support supported10+ primary 6912 562 0.313624 0.838623 6 khoa_quan_tri_kinh_doanh_va_du_lich course_support supported10+ committee 6912 562 0.390272 0.857690 6 khoa_quan_tri_kinh_doanh_va_du_lich student_identity no_recorded_history primary 12 5 0.549394 0.371429 6 khoa_quan_tri_kinh_doanh_va_du_lich student_identity no_recorded_history committee 12 5 0.597778 0.714286 6 khoa_quan_tri_kinh_doanh_va_du_lich student_identity recorded_history primary 7430 588 0.307777 0.841012 6 khoa_quan_tri_kinh_doanh_va_du_lich student_identity recorded_history committee 7430 588 0.376065 0.857807 6 khoa_quoc_te_hoc history 1-4 primary 53 1 0.100000 0.826923 6 khoa_quoc_te_hoc history 1-4 committee 53 1 0.125000 0.865385 6 khoa_quoc_te_hoc history 5-16 primary 1421 50 0.164509 0.795441 6 khoa_quoc_te_hoc history 5-16 committee 1421 50 0.343815 0.800620 6 khoa_quoc_te_hoc history 17+ primary 992 9 0.048760 0.740590 6 khoa_quoc_te_hoc history 17+ committee 992 9 0.061923 0.819261 6 khoa_quoc_te_hoc course_support unseen primary 226 6 0.258913 0.758712 6 khoa_quoc_te_hoc course_support unseen committee 226 6 0.364743 0.680682 6 khoa_quoc_te_hoc course_support rare1-9 primary 9 0 NaN NaN 6 khoa_quoc_te_hoc course_support rare1-9 committee 9 0 NaN NaN 6 khoa_quoc_te_hoc course_support supported10+ primary 2231 54 0.124589 0.813105 6 khoa_quoc_te_hoc course_support supported10+ committee 2231 54 0.263172 0.841423 6 khoa_quoc_te_hoc student_identity recorded_history primary 2466 60 0.123194 0.807332 6 khoa_quoc_te_hoc student_identity recorded_history committee 2466 60 0.277844 0.823937 6 khoa_tieng_anh history 0 primary 136 7 0.057488 0.506091 6 khoa_tieng_anh history 0 committee 136 7 0.082026 0.635659 6 khoa_tieng_anh history 1-4 primary 1011 26 0.105421 0.826982 6 khoa_tieng_anh history 1-4 committee 1011 26 0.209713 0.881179 6 khoa_tieng_anh history 5-16 primary 3197 52 0.221115 0.838810 6 khoa_tieng_anh history 5-16 committee 3197 52 0.360668 0.794231 6 khoa_tieng_anh history 17+ primary 1860 37 0.166244 0.825251 6 khoa_tieng_anh history 17+ committee 1860 37 0.326000 0.862626 6 khoa_tieng_anh course_support unseen primary 734 9 0.240526 0.816092 6 khoa_tieng_anh course_support unseen committee 734 9 0.327699 0.810958 6 khoa_tieng_anh course_support rare1-9 primary 26 2 0.583333 0.958333 6 khoa_tieng_anh course_support rare1-9 committee 26 2 0.118056 0.520833 6 khoa_tieng_anh course_support supported10+ primary 5444 111 0.140204 0.823099 6 khoa_tieng_anh course_support supported10+ committee 5444 111 0.285615 0.829148 6 khoa_tieng_anh student_identity no_recorded_history primary 136 7 0.057488 0.506091 6 khoa_tieng_anh student_identity no_recorded_history committee 136 7 0.082026 0.635659 6 khoa_tieng_anh student_identity recorded_history primary 6068 115 0.161066 0.830769 6 khoa_tieng_anh student_identity recorded_history committee 6068 115 0.297530 0.825553 6 khoa_tieng_bo_dao_nha history 1-4 primary 180 30 0.515093 0.844000 6 khoa_tieng_bo_dao_nha history 1-4 committee 180 30 0.636773 0.879333 6 khoa_tieng_bo_dao_nha history 5-16 primary 266 24 0.335816 0.810864 6 khoa_tieng_bo_dao_nha history 5-16 committee 266 24 0.568302 0.838499 6 khoa_tieng_bo_dao_nha history 17+ primary 446 19 0.212719 0.870701 6 khoa_tieng_bo_dao_nha history 17+ committee 446 19 0.313381 0.882288 6 khoa_tieng_bo_dao_nha course_support unseen primary 145 18 0.584883 0.865923 6 khoa_tieng_bo_dao_nha course_support unseen committee 145 18 0.671101 0.859580 6 khoa_tieng_bo_dao_nha course_support rare1-9 primary 25 0 NaN NaN 6 khoa_tieng_bo_dao_nha course_support rare1-9 committee 25 0 NaN NaN 6 khoa_tieng_bo_dao_nha course_support supported10+ primary 722 55 0.426045 0.872155 6 khoa_tieng_bo_dao_nha course_support supported10+ committee 722 55 0.528157 0.894071 6 khoa_tieng_bo_dao_nha student_identity recorded_history primary 892 73 0.384397 0.856499 6 khoa_tieng_bo_dao_nha student_identity recorded_history committee 892 73 0.521049 0.880593 6 khoa_tieng_duc history 0 primary 14 1 1.000000 1.000000 6 khoa_tieng_duc history 0 committee 14 1 0.166667 0.615385 FROZEN HYPOTHESIS AUDIT -------------------------------------------------------------------------------- primary contextual fulltab_hgb mirt committee suite22_blend suite25_blend contextual_increment bottleneck_increment beats_2026_reference_by_0p02 reference_coverage_note 0.286328 0.246353 0.343467 0.291853 0.350152 0.369159 0.369526 0.039975 -0.005525 False Compare aligned cells, not unmatched aggregates. Historical F6/F7 previously used for design. [DONE] Suite28 finished elapsed_hours=1.913 results=/root/disk_3tb/earlywarning/30072026/results/suite28_dynamic_readiness_requirement [COMPLETE] 2026-10-08T12:26:46+00:00 Suite28 DRRM