# Suite26 MINT — complete joint tree–memory experiment

Primary prediction column: **`suite26_mint`**. Memory participates in each internal tree decision; the forest's trainable leaves produce the final probability. Read `DESIGN.md` for the exact architecture and `REVIEW_AND_RATIONALE.md` for the assessment of the original result.

This package is a new experiment. It preserves the original Suite26 code/results by using a separate default output directory. It cannot guarantee a performance improvement. The included validation is CPU execution on synthetic data, not an AQ or V100 benchmark.

## Run on the existing AQ server

Extract the archive under `/root/disk_3tb/earlywarning/30072026`, then:

```bash
cd /root/disk_3tb/earlywarning/30072026/suite26_joint_tree_memory
bash run_suite26.sh --preflight-only
bash run_suite26.sh
```

The runner uses `$PYTHON` when supplied, otherwise `/root/miniconda3/envs/Huy/bin/python3` when available, otherwise `python3`. It does not install or upgrade dependencies. The previous log's PyTorch 2.7.1 + CUDA 11.8 environment satisfies the stated package versions; execution on that environment remains to be verified.

Default input locations:

```text
/root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache
/root/disk_3tb/earlywarning/30072026/results/suite12_structured_cache
```

Default output:

```text
/root/disk_3tb/earlywarning/30072026/results/suite26_joint_tree_memory
```

Defaults run all discovered faculties and folds 1–7. Folds 1–5 supply operational thresholds; folds 6–7 are historical comparison folds. Epoch selection uses the latest **one earlier historical semester**, never the target fold. A global calendar cutoff applies across faculties. Every model bundle is saved.

Optional logging:

```bash
set -o pipefail
bash run_suite26.sh 2>&1 | tee ../suite26_mint.log
```

Inspect all options with `bash run_suite26.sh --help`. Flags accept space-separated or comma-separated fold/faculty lists. Example overrides:

```bash
PYTHON=/root/miniconda3/envs/Huy/bin/python3 bash run_suite26.sh \
  --root /root/disk_3tb/earlywarning/30072026 \
  --faculties khoa_cong_nghe_thong_tin khoa_quan_tri_kinh_doanh_va_du_lich \
  --folds 1 2 3 4 5 6 7 \
  --out-dir /root/disk_3tb/earlywarning/30072026/results/suite26_mint_two_faculties
```

`--cache-dir`, `--structured-dir`, `--device`, and `--out-dir` can be set independently. Use `--device cpu` for CPU execution. Reduce `--batch-size` and `--eval-batch-size` if device memory is insufficient. Preflight checks the same requested cache/schema/chronology settings without creating results; it is not a GPU allocation benchmark.

## What is trained

| Column | Purpose |
|---|---|
| `suite26_seed_forest` | Conventional initialization, with natural-count leaf probabilities |
| `suite26_tree_only` | Jointly trained forest with current/course context |
| `suite26_memory` | Same forest with node-conditioned episodic memory |
| `suite26_mint` | Same forest with episodic memory and readiness conditioning |

All learned variants begin with the same tree blueprint and parameter RNG seed. They share their maximum selection budget, data access for current features, and feature dimensions; their epoch counts are selected separately from historical validation. Refit uses fresh forests/models and preserves each selection-stage examples-per-epoch budget. Historical raw features can already encode past performance, so `tree_only` is a control for **additional episodic memory**, not a history-free student predictor.

The seed uses balanced weights to discover splits, then replaces leaf scores with **unweighted counts** plus 0.5/0.5 smoothing. Neural BCE uses no class weights. Faculty-balanced sampling preserves the class distribution within each faculty in expectation. Optional ranking has weight 0.05; use `--rank-weight 0` for the pure BCE objective.

The default forest has 64 trees, depth 6, hidden width 32, 16 readiness dimensions, and at most 32 recent historical events per query. This geometry controls the cost of dense node attention. It is smaller than the original 96-tree, 768-leaf-per-tree forest and is not claimed to match the Committee's capacity. A larger development-only capacity experiment can use `--trees 96 --depth 8 --batch-size 64 --eval-batch-size 64` with a new output directory. Do not pick geometry from F6/F7 performance and then treat those folds as untouched tests.

## Temporal and cache contract

Canonical `<faculty>_fold_<fold>.joblib` files must include `feature_names`, `Xtr`, `Xte`, `ytr`, `yte`, `train_frame`, and `test_frame`. Frames require `IDSV`, `IDMH`, and `target_NHHK` (year*10 + term, with terms 1–3). Historical `target_grade10` is optional; known grades must lie in 0–10 and agree with failure `<5`. Missing grades use the observed binary outcome for readiness. Target grades are discarded from model data. Committee probabilities are a reporting reference only.

Suite12 files require `train_row_index`, `train['course_target']`, and `target['course_target']`, each with the canonical 17 course fields. The loader respects reordered train indices. Target order follows the canonical cache unless an explicit `target_row_index` permutation is supplied. The complete raw-feature schema is the intersection across **all requested faculties and folds**.

All canonical historical rows are retained, including rows without CF coverage. Uncovered rows receive earlier-semester course statistics; missing static metadata remains explicitly missing. Course IDs and student IDs are presumed globally unique in the original data. Course vocabulary, scaling and imputation use the permitted fitting rows only.

History contains only the student's strictly earlier-semester events. Same-semester outcomes are excluded, including when there are multiple courses. Candidate selection uses recency, not outcomes. Grades are assumed available by the following semester because caches have no release timestamps. Arbitrary precomputed raw/covered features remain trusted as-of-query inputs: their numeric arrays cannot prove provenance. `--val-semesters > 1` prints and saves the distinction between fixed-origin episodic memory and row-time current features that may use earlier validation-semester outcomes. The default one-semester validation avoids this reconstruction-specific mismatch.

## Results and interpretation

Top-level outputs include predictions, per-cell/per-fold/per-faculty metrics, macro metrics, paired comparisons, component uplifts, thresholds, operational metrics, IT confusion matrices, bootstrap intervals/replicates, and keyed reference-alignment audits.

Each `fold_<n>/` contains:

- `tree_only.pt`, `memory.pt`, `mint.pt`: complete reloadable bundles with preprocessing, course vocabulary, blueprint, state and provenance.
- `training_log.csv`: loss, sampled faculty counts, ranking participation, validation AP, time/RSS/CUDA allocation and **main-BCE** gradient norms.
- `data_audit.json`, `fold_summary.json`, `tree_diagnostics.json`: coverage, chronology, chosen epochs/example budgets, feature-choice changes, base-threshold drift and temperatures.
- `memory_diagnostics.csv`, `regime_metrics.csv`: history support, unknown courses, attention summaries, readiness/gap and threshold shifts.
- `intervention_predictions.csv`, `intervention_metrics.csv`, `intervention_macro_ap.json`: memory/outcome/readiness masking, uniform attention, hard routing, no context, shuffled students and history-K sensitivity.
- `history_shuffle_audit.json`, `complete.json`: donor/time audit and completed-output hashes.

Intervention scores are post-training sensitivities, not separately trained ablations or causal estimates. `no_outcomes` masks direct historical grades/failure labels and readiness conditioning; precomputed raw features may still contain indirect past-outcome information. `no_memory` removes both episodic retrieval and readiness conditioning, while `no_ability` removes readiness only. `no_context` retains learned base splits/leaves and removes every conditional threshold term. Hard routing is a diagnostic; normal training and prediction use the same soft routing.

A fair positive result requires MINT to beat the **jointly trained tree-only control** on matched rows, with its component benefits consistent enough across faculties/folds and uncertainty to support the claim. Positive memory gradients, route changes, or a harmful shuffle establish computational use; none establishes improved discrimination. Also compare against the strongest aligned prior model, inspect calibration and the IT confusion matrix, and include losses as well as wins.

Prior suites are discovered automatically under the root's results directory. Original scores are mapped to `suite26_original`, `suite26_original_memory`, `suite26_original_tree`, and `suite26_original_joint_head`. Use explicit mappings such as `--reference-dirs suite22=/path/to/suite22 suite26_original=/path/to/original`. Passing `--reference-dirs` with no values disables external references. Files need one-to-one identifiers; ambiguous or conflicting rows are rejected with a reason, and partial coverage stays missing. Compare **paired** metrics when faculty/row coverage differs. Old files with only matching label arrays are deliberately insufficient.

## Resume and reload

A nonempty result directory is refused unless `--resume` is supplied. Resume requires the exact code, dependency versions, training configuration and input-cache hashes; completed output hashes are verified. Partial folds are restarted, not continued at a minibatch. Choose a new result directory after changing settings or source.

```bash
bash run_suite26.sh --resume

/root/miniconda3/envs/Huy/bin/python3 predict_suite26.py \
  --checkpoint /root/disk_3tb/earlywarning/30072026/results/suite26_joint_tree_memory/fold_7/mint.pt \
  --out-file /root/disk_3tb/earlywarning/30072026/suite26_mint_reloaded_fold7.csv
```

Reload refuses changed inference cache hashes unless explicitly permitted with `--allow-changed-cache`, and refuses an existing prediction output file. Checkpoints contain pickle-backed NumPy state; load only your own trusted experiment bundles.

## Software verification

Use the same Python environment as training:

```bash
python3 -m unittest discover -s tests -v
python3 smoke_test.py --out-dir /tmp/suite26_mint_smoke
```

The smoke directory must be new or empty. It exercises two canonical folds, all learned variants, selection/refit, interventions, reports and checkpoint replay. The unit suite additionally tests chronology, missing/unknown courses, grade consistency, padding/permutation invariance, hard seed conversion including adjacent float32 boundaries, main-loss gradients for tree/memory/readiness, target-label independence, reference alignment, sampling-budget parity, resume and mismatch rejection.

`validation/` contains the execution evidence and original-log summary extraction. `VALIDATION.md` distinguishes what was executed from what remains untested. `review_original_log.py` can reproduce the original printed-summary extraction; it does not reconstruct missing row predictions.
