# Suite29 v3.1 — Semester-Conditioned Enrollment Set Learning

## Status

**Source-level audit completed; synthetic integration run and checkpoint replay passed.**
The full AQ dataset is not included and the real 14-faculty run has **not** been executed locally. Do not interpret synthetic metrics as real AQ performance.

## Hypothesis and unit of prediction

For one student `s`, one **known** semester `t`, and one faculty, gather all observed registrations into an unordered set `C(s,t)`. Each course retains its own per-registration features. One shared encoder transforms the set into tokens. Masked multi-head self-attention allows each real course token to attend to other real course tokens, followed by a residual feed-forward network and **one logit for each individual course**.

The model **does not** predict one aggregate probability for the semester and it **does not** ingest concurrent course grades. It predicts each course's failure using the composition of the *known upcoming registrations*. The attention module is a single Set-Transformer-style block (not a full ISAB/PMA hierarchy). No positional embeddings are used, preserving permutation equivariance.

**Variants, all with one coherent prediction boundary:**
- `suite29_self_only`: separately trained architecture-matched control, permitting attention to the query itself and a learned null token but not to peers.
- `suite29_set`: separately trained primary model, permitting attention to every valid same-student/same-semester/same-faculty course and the learned null token.
- `suite29_set_no_peers`: **test-time intervention** on the separately trained `set` model. Not a separately trained model.
- `committee`: archived reference only; never used as a model input or combined with Suite29 predictions.

Both trainable variants have identical architectures/parameter counts and use the same seed, training rows, early-stopping rules and optimizer configuration. Their fitted parameters and selected epochs may differ, as intended.

## Actual input contract

Adapted from the original `suite25_structure_aware_chronological_retrieval` source. Bundles use:

- `results/cross_faculty_residual_moe/cache/<faculty>_fold_<n>.joblib` (canonical cache)
- `results/suite12_structured_cache/<faculty>_fold_<n>.joblib` (structured target-course features)
- The **intersection** of raw feature names in the selected faculties, ordinarily **136**, plus exactly **17 structured target-course variables**, ordinarily **153** in total.
- Group keys `(faculty, IDSV, target_NHHK)`; the grouping key is **not** a learned student-ID embedding.

`common raw features` means a shared feature **schema**. They are not assumed constant within a student semester. A faculty-specific registration cache may omit the student's simultaneous registrations elsewhere in the university; in that case Suite29 sees a **partial observed enrollment set**.

The canonical history rows are restricted by the previous suites' chronological cross-fitting coverage. `all_history_coverage_audit.csv` now compares retained historical groups with full canonical historical registrations and reports partial groups, which could cause distribution shift between historical training and target. `--strict-complete-history-groups` can fail instead of proceeding if this occurs. The standard run audits and reports it without dropping rows automatically, because dropping selected rows silently would change the historical experiment population.

The provenance of **all upstream engineered features** is not fully certified by these source packages. This experiment enforces its own chronology but cannot retroactively certify the origin of every raw or engineered input. In particular, any column derived after a prediction cutoff must be removed upstream before interpreting the results prospectively.

## Temporal evaluation

The original v1 code selected the latest validation blocks separately in each faculty. **v2 fixes that potential cross-faculty label-timing contamination.** A single global calendar-semester cutoff is now used for each historical selection run, based on `target_NHHK` converted to an academic semester ordinal. Training semesters strictly precede **all** validation semesters in the shared network. Output includes the cutoff and training/validation row counts per faculty. For target refit, v2 also **fails rather than proceeding** if the latest training-semester ordinal is at or after the earliest target-semester ordinal across faculties.

The outer target is never used for epoch selection. After selecting an epoch on historical training/validation, a freshly initialized model is refitted for that number of epochs on permissible historical rows, and only then predicts its target fold. All results for all 14 faculties and F1–F7 can be generated.

**Important:** The historical project has repeatedly used F6/F7 for exploration. They are *retrospective* comparisons, not a pristine independent holdout; new prospective data would be required for definitive confirmation.

## Installation and validation

Requires **Python >=3.10**, `numpy`, `pandas`, `joblib`, `scipy`, `scikit-learn`, `torch`. Use the existing AQ environment where Suite25 ran.

```bash
unzip suite29_enrollment_set_v2.zip -d /root/disk_3tb/earlywarning/30072026/
cd /root/disk_3tb/earlywarning/30072026/suite29_enrollment_set
/root/miniconda3/envs/Huy/bin/python3 -m unittest discover -s tests -v
```

### One-faculty smoke test on your actual server caches

This is to verify paths, chronology, memory, and the actual feature contract; **do not treat it as the final cross-faculty metric** (the common feature intersection is different).

```bash
/root/miniconda3/envs/Huy/bin/python3 -u train_suite29.py \
  --folds 1 --faculties khoa_cong_nghe_thong_tin \
  --seeds 29 --epochs 2 --group-batch-size 32 \
  --output-dir /root/disk_3tb/earlywarning/30072026/results/suite29_smoke
```

### Full cross-faculty experiment

```bash
/root/miniconda3/envs/Huy/bin/python3 -u train_suite29.py \
  --cache-dir /root/disk_3tb/earlywarning/30072026/results/cross_faculty_residual_moe/cache \
  --structured-dir /root/disk_3tb/earlywarning/30072026/results/suite12_structured_cache \
  --output-dir /root/disk_3tb/earlywarning/30072026/results/suite29_enrollment_set \
  --folds 1,2,3,4,5,6,7 --faculties all \
  --modes self_only,set --seeds 29,129,229 \
  --epochs 18 --patience 5 --validation-blocks 3 \
  --group-batch-size 64 --save-checkpoints
```

Defaults are identical unless overridden. If any **global target-chronology check** fails, do not bypass it by changing the feature file names: inspect the target and history periods and adjust the data population/experiment contract safely.

You can alternatively use `PYTHON=/root/miniconda3/envs/Huy/bin/python3 bash run_suite29.sh [arguments]` to run unit tests first and capture training output in a timestamped log.

### Replaying a saved checkpoint

Saved checkpoints include the exact model parameters, raw input-feature order, faculty order, fitted means/scales, semester normalization, selected epoch and model hyperparameters; these were missing in v1.

```bash
/root/miniconda3/envs/Huy/bin/python3 verify_checkpoint.py \
  --checkpoint /root/disk_3tb/earlywarning/30072026/results/suite29_enrollment_set/fold1_set_seed29.pt \
  --scores /root/disk_3tb/earlywarning/30072026/results/suite29_enrollment_set/fold_1_seed_29_scores.csv
```

Use only checkpoints you trust; checkpoint loading relies on PyTorch object deserialization. Exact-row replay also depends on having the same underlying cache files.

## Output and interpretation

| File | Explanation |
| --- | --- |
| `fold_<fold>_seed_<seed>_scores.csv` | All original target rows in order, original outcome, registration/student/course/semester IDs, observed set size, Committee, both primary variants, peer-ablated set prediction, prediction deltas |
| `all_metrics.csv` | AP, AUC, Brier, ECE10, precision/recall/F2 at alert budgets 10%, 15%, 20%, 25%, for each faculty/fold/seed/model |
| `paired_set_vs_self.csv` | Paired faculty/fold/seed differences in AP, AUC, F2@20% |
| `all_selection_curves.csv` | Historical selection macro AP/F2/Brier and training loss for every epoch |
| `all_selection_faculty_curves.csv` | Historical selection AP/AUC/F2/Brier for each faculty and epoch |
| `all_attention_diagnostics.csv` | Whole-fold attention mass to self, peers, learned null |
| `all_attention_stratified.csv` | Attention mass stratified by faculty, enrollment-set size bin and observed fail/pass |
| `all_group_diagnostics.csv` | Student–semester group-level observed and expected failures, count error, Brier and approximate any-failure probability |
| `all_group_size_summaries.csv` | Aggregated group count/calibration/error by observed set-size band |
| `all_history_coverage_audit.csv` | Cross-fitting row coverage and partial historical sets per faculty/fold |
| `all_bootstrap.csv` | Descriptive paired AP intervals from within-faculty student-cluster bootstrap |
| `coverage_manifest.csv` | Flags missing faculty/fold coverage to prevent treating partial baselines as a complete comparison |
| `macro_all_folds.csv` | Descriptive macro averages, all folds |
| `macro_folds_6_7_retrospective.csv` | Descriptive macro averages for repeated F6/F7 comparisons |
| `fold<fold>_<mode>_seed<seed>.pt` | Full checkpoint **with preprocessing** if `--save-checkpoints` is enabled |
| `config.json`, `features.json`, `runtimes.csv` | Reproducibility and runtime metadata |

**Primary success criterion**: consistently positive `suite29_set - suite29_self_only` faculty–fold AP on chronological test targets. Attention mass by itself is **not** evidence of predictive benefit or causal importance. The `set_no_peers` intervention is a separate mechanistic diagnostic, not a substitute for the separately trained control. Set-size-specific results must be checked because no peer benefit is possible in singleton groups.

Averages of archival reference models with **incomplete** coverage may not be compared naively. The `coverage_manifest.csv` marks their coverage. Optional `--reference name=/path` loading checks lengths and outcomes; without an exact row identifier in the reference file this alone does **not** prove row alignment.

## Local tests carried out by the package author

- 12 unit tests: permutation equivariance, padding invariance, singleton equivalence, peer independence in the self-only control, peer attention, gradients, grouping, global calendar cutoff and failure modes, stratified attention, padding/batch consistency, historical partial-coverage detection.
- Two-faculty synthetic-cache end-to-end runner test, including selection, refit, row-aligned prediction generation, coverage and per-group reporting.
- Saved checkpoint replay for both separately trained variants; agreement with saved predictions on synthetic data to numerical precision.
- **No real all-faculty AQ run** has been executed in this environment; downstream source-provenance and feature timing remain open verification tasks.

---

## Suite29 v3 launcher and reporting update (compared with v2)

The **recommended entry point is `bash run_suite29.sh`**, rather than calling the
Python training script directly. It now follows the filesystem conventions of
Suites 24, 25, 26, and 28.

### Canonical server layout

```text
/root/disk_3tb/earlywarning/30072026/
├── suite29_enrollment_set/                 # Extracted source-code package
│   ├── run_suite29.sh                      # Run this (environment → tests → training)
│   ├── check_suite29_environment.py        # File/dependency preflight
│   ├── train_suite29.py                    # Unchanged primary architecture/training logic
│   ├── suite29_report.py                   # Final cross-fold research summary
│   └── ...
├── log/
│   ├── suite29_enrollment_set.log          # Stable, append-only log across runs
│   └── suite29_enrollment_set_<timestamp>_<pid>.log # Isolated log for each run
└── results/
    ├── suite29_enrollment_set/             # Full experiment outputs + checkpoints
    └── suite29_smoke/                      # Recommended separate smoke-test outputs
```

`run_suite29.sh` defaults to `/root/miniconda3/envs/Huy/bin/python3`. Override
this with `PYTHON=...`, or override the project root with `AQ_ROOT=...`.

**Full run (no arguments needed):**

```bash
cd /root/disk_3tb/earlywarning/30072026/suite29_enrollment_set
bash run_suite29.sh
```

This runs **all seven folds**, **all faculties**, **both variants**,
**three seeds (29,129,229)**, **up to 18 epochs**, and saves checkpoints.
The launcher's explicit defaults can be overridden by repeating a CLI option:

```bash
bash run_suite29.sh \
  --folds 1 \
  --faculties khoa_cong_nghe_thong_tin \
  --seeds 29 \
  --epochs 2 \
  --group-batch-size 32 \
  --bootstrap 20 \
  --output-dir /root/disk_3tb/earlywarning/30072026/results/suite29_smoke
```

**Please use a different output directory for the smoke test** so that the
full-run results are not contaminated with older partial files or checkpoints.
Do not interpret the 2-epoch smoke run as a meaningful model comparison.

To inspect the current run's log:

```bash
tail -f /root/disk_3tb/earlywarning/30072026/log/suite29_enrollment_set.log
```

The stable log is append-only like Suite28. Each run has a timestamped copy,
so later launches cannot overwrite the complete log from an earlier run.
The launcher's exit code reflects preflight, unit-test, and training failures.
It prints `[RUN STATUS] SUCCESS` or `[RUN STATUS] FAILURE` accordingly.

### Reported insights (also saved as CSV)

At each fold and seed, the runner prints per-faculty AP/AUC, prevalence,
precision/recall/F2 at 20% warning budget, Brier, ECE, enrollment-set
composition, expected versus observed failure count, and self/peer/null
attention by faculty and observed enrollment-set size. Chronological selection
prints train/validation coverage, epoch-by-epoch macro AP/F2/Brier, selected
epoch, and refit losses.

The final `suite29_report.py` prints and saves:

- `summary_fold_seed_macro.csv` — macro AP/AUC/F2/Brier by fold, seed and model;
- `summary_retrospective_f6_f7.csv` — clearly marked retrospective F6/F7 macro;
- `summary_reference_coverage.csv` — guards against ranking partial baselines;
- `summary_paired_all.csv`, `summary_paired_wins.csv`,
  `summary_paired_by_faculty.csv`, `summary_paired_retrospective_f6_f7.csv` —
  Set-minus-self deltas, wins/losses, and stability;
- `summary_set_attention_faculty_size.csv` — token attention allocation;
- `summary_group_size_failure_counts.csv` — predicted versus observed counts;
- `summary_history_partial_groups.csv` — incomplete history-group warnings;
- `summary_bootstrap.csv` — descriptive student-cluster uncertainty.

`all_target_enrollment_profiles.csv` is also added to record target group counts,
singletons, course-set size, and observed multiple-failure-group prevalence.
These are **descriptive outcomes for analysis**, never model inputs.

To regenerate the final analysis from completed CSVs without retraining:

```bash
/root/miniconda3/envs/Huy/bin/python3 suite29_report.py \
  --output-dir /root/disk_3tb/earlywarning/30072026/results/suite29_enrollment_set
```

`committee` is included by default. Additional Suite22/Suite25 historical
predictions can be loaded with `--reference NAME=/path` (repeatable), provided
per-faculty row alignment and outcome correspondence are verified. The runner
does not automatically fabricate reference paths or missing model predictions.

**Research interpretation:** The full model is unchanged: one set of concurrently
enrolled courses goes through the masked set-attention block, yielding one
failure probability *per course*. The self-only variant remains a matched
control; the no-peers mode is a test-time intervention. Attention allocation
is not a causal explanation. Full upstream feature as-of provenance and real
14-faculty execution still require verification on the university server.


## v3.1 real-cache smoke test hotfix (2026-10-09)

- Fixed a NumPy `int8` overflow in the `[DATA]` history/target failure-count print.
  The earlier smoke run printed impossible values, including `failures=-67`, while
  the independent target evaluation correctly counted 189/5222 failures.
- Added binary-label validation and an `int8` count regression test.
- Added a warning when a subset-of-faculties smoke run selects a different raw
  feature intersection from the full model. The smoke run has 275+17=292
  features; the historical full-faculty intersection is ordinarily 136+17=153.
- No change to grouping, model, loss, optimizer, epoch selection, refit,
  predictions, or output schema. Version v3.1 is a logging/validation hotfix.

**Interpret the 2-epoch Fold 1 IT smoke run as integration validation only,
not as evidence that the enrollment-set hypothesis has failed.**
