# Suite31 — CAPT (Chronologically Adapted Pretrained Tabular Transfer)

**Status: source release, not a claim of completed V100/AQ training.** This package implements the Suite31 experiment machinery and passes the included standard-library/unit and synthetic HGB end-to-end checks. The local release environment **does not have the TabICLv2/TabPFN packages or checkpoints**, so actual external-weight inference and fine-tuning must be validated on your research server. These branches fail explicitly if optional model packages are missing. No dummy predictions are substituted for foundation-model results.

## Scientific objective

Test whether an **external pretrained tabular prior** improves student-course failure prediction on the existing AQ 136 shared raw + 17 course features, then whether **time-ordered historical support → later-semester query** fine-tuning beats ordinary AQ randomized fine-tuning. Do **not** infer that results will outperform the Committee or Suite25 blend. Suite29–30 is retired as the main performance direction.

- `tabicl_frozen`: externally pretrained in-context inference, no AQ gradient updates.
- `tabpfn_35_frozen`: independent local TabPFN comparator (default currently 3.5; verify actual installed model/version).
- `hgb_matched`: identical faculty-local sampled support as pretrained candidates, natural failure prevalence.
- `hgb_full`: all eligible faculty-local historical support.
- `committee`: existing exact target committee score from the canonical cache, reference only.
- `tabicl_ordinary`: real upstream random-context/query fine-tuning, externally pretrained initialization, **historical faculty–origin macro-AP** validation.
- `tabicl_chronological`: real upstream TabICL neural weights optimized with *explicit historical* temporal context/query batches, historical macro-AP validation.

Use the same prediction-time support size, feature contract and deterministic support identifiers for model comparisons. An origin's own labels never enter its support. F6/F7 are repeatedly consulted **retrospective** evaluations, not untouched prospective tests.

## 0. Installation (on the V100)

```bash
unzip suite31_CAPT_complete.zip -d /root/disk_3tb/earlywarning/30072026/
cd /root/disk_3tb/earlywarning/30072026/suite31_CAPT
conda activate Huy   # or the environment containing your working Torch/CUDA installation
python -m pip install -r requirements.txt
python -m pip install 'tabicl[finetune]'
python -m pip install tabpfn # Optional, non-commercial weight license; local research only
```

Check `torch.cuda.is_available()` on your host and ensure V100-compatible PyTorch builds. The upstream checkpoint will be downloaded on first use unless `--tabicl-checkpoint` / `--tabpfn-checkpoint` points to locally cached models. External weights are not included in this ZIP.

The upstream TabICL source tested by code review is `https://github.com/soda-inria/tabicl` (`src/tabicl/_finetune/base.py`, `data.py`); this release checks for the required source entry points at runtime and fails if they are missing. **Pin your resolved pip versions after your first successful real integration smoke; do not silently update midway through evaluation.** TabPFN-3.5 weights have licensing conditions distinct from the Apache-licensed code. Do not use a hosted model with identifiable student data.

## 1. Preflight and unit tests

```bash
export SUITE31_ROOT=/root/disk_3tb/earlywarning/30072026
bash run_suite31.sh --stage preflight --folds 1 --faculties all
PYTHONPATH="legacy:tests:." python -m unittest discover -s tests -p 'test_suite31*.py' -v
```

**Preflight checks source data alignment/feature continuity, not actual grade-release timestamps.** Cache `target_NHHK` semester chronology is a proxy. Use `--release-gap 1` as a deliberate sensitivity test; an embargo does not prove timestamp availability. The run emits a warning even when the chronology audit passes.

Run dependency-free synthetic end-to-end check:

```bash
SUITE31_ROOT=/tmp/suite31_synthetic bash run_suite31.sh --stage synthetic \
  --device cpu --models committee,hgb_matched,hgb_full --support-budgets 40,100 --seeds 29
```

**Synthetic Committee probabilities are generated from labels by the Suite30 fixture and are unrealistically strong. Never interpret synthetic AP as model evidence.**

## 2. V100 model integration smoke

Start with one manageable faculty, one fold, one seed and reduced support. This tests *actual* local pretrained weights and GPU memory before the full experiment.

```bash
bash run_suite31.sh --stage pilot \
  --faculties khoa_cong_nghe_thong_tin --folds 1 --seeds 29 \
  --support-budgets 512 --models hgb_matched,hgb_full,tabicl_frozen \
  --device cuda --query-batch-size 128 --n-estimators 2 \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_model_smoke"
```

TabPFN smoke (explicitly check installed default version is 3.5):

```bash
bash run_suite31.sh --stage resource \
  --faculties khoa_cong_nghe_thong_tin --folds 1 --seeds 29 \
  --support-budgets 512,2048 --device cuda --query-batch-size 128 \
  --n-estimators 2 --output-dir "$SUITE31_ROOT/results/suite31_CAPT_resource"
```

The first real foundation-model smoke is **required**. Local release tests did not execute against downloaded checkpoints.

## 3. Gate 1 — Frozen transfer pilot

Run only the predeclared three faculties, two context sizes, three seeds; substitute resource-feasible budgets before measuring AP if 16384 cannot run on the V100.

```bash
bash run_suite31.sh --stage pilot \
  --faculties khoa_cong_nghe_thong_tin,khoa_quan_tri_kinh_doanh_va_du_lich,khoa_tieng_trung_quoc \
  --folds 1,2,3 --seeds 29,129,229 --support-budgets 4096,16384 \
  --models committee,hgb_matched,hgb_full,tabicl_frozen,tabpfn_35_frozen \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_pilot"
```

Comparisons are within exact canonical target rows and the same support sampling manifest. Support prevalence is *not* balanced to 50/50. A single-class context uses an explicitly printed, smoothed historical prior fallback across supported models.

## 4. Gate 2 — Ordinary versus chronological fine-tuning

**Ordinary fine-tuning** uses upstream randomized meta-batches, capped to the same nominal optimizer steps as the chronological arm. Both select checkpoints on chronological historical macro AP.

```bash
bash run_suite31.sh --stage ordinary_finetune \
  --folds 1 --faculties all --seeds 29 --support-budgets 4096 \
  --models tabicl_frozen,tabicl_ordinary,hgb_full,committee \
  --epochs 2 --patience 1 --episode-support-size 512 \
  --episode-query-size 64 --max-episodes 8 \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_ordinary_smoke"
```

**Chronological CAPT** uses real `MetaBatch` tensors built with upstream `EnsembleGenerator`, with contexts strictly preceding historical query origins. It does not simply sort rows and invoke ordinary `.fit`.

```bash
bash run_suite31.sh --stage chronological_adapt \
  --folds 1 --faculties all --seeds 29 --support-budgets 4096 \
  --models tabicl_frozen,tabicl_chronological,hgb_full,committee \
  --epochs 2 --patience 1 --episode-support-size 512 \
  --episode-query-size 64 --max-episodes 8 \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_chronological_smoke"
```

**Resource warning:** in-context validation can be expensive. Start with small contexts/epochs, confirm actual memory and runtime, and scale experimentally rather than launching all folds × seeds × models immediately. Common `--max-episodes` controls cap gradient updates. The ordinary arm and chronological arm still differ in which historical rows they expose per update; record this caveat when interpreting their difference.

For the full all-faculty *development* study, only use folds 1–5. Running the identical adaptation settings in both arms yields an internally consistent comparator:

```bash
bash run_suite31.sh --stage development --folds 1,2,3,4,5 --faculties all \
  --models committee,hgb_matched,hgb_full,tabicl_frozen,tabpfn_35_frozen,tabicl_ordinary,tabicl_chronological \
  --seeds 29,129,229 --support-budgets 4096,16384 \
  --epochs 4 --patience 2 --episode-support-size 1024 \
  --episode-query-size 128 --max-episodes 64 \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_development"
```

This emits `frozen_development_protocol.json` only if all requested configurations complete without errors. **This is an experimental protocol lock, not proof of efficacy or proper prospective isolation.** The lock guards key settings, not every possible implementation/source hash.

## 5. Gate 3 — F6/F7 retrospective

Only after choosing the configuration on historical development and generating its protocol lock. Change the `--models` list and other options to **exactly** what the development lock records.

```bash
bash run_suite31.sh --stage retrospective --folds 6,7 --faculties all \
  --models committee,hgb_matched,hgb_full,tabicl_frozen,tabpfn_35_frozen,tabicl_ordinary,tabicl_chronological \
  --seeds 29,129,229 --support-budgets 4096,16384 \
  --epochs 4 --patience 2 --episode-support-size 1024 \
  --episode-query-size 128 --max-episodes 64 \
  --protocol-freeze-file "$SUITE31_ROOT/results/suite31_CAPT_development/frozen_development_protocol.json" \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_retrospective"
```

For F6/F7, the algorithm/training policy is frozen but the model is fitted using **only its own preceding historical semesters**. Therefore this is expanding-origin retrospective fitting, *not* inference with one single fixed checkpoint from the development folds. It is important to preserve this distinction in the paper.

## 6. Logs and outputs

Stable shell log: `$SUITE31_ROOT/log/suite31_CAPT.log`. Launcher propagates nonzero exits through `set -euo pipefail` and prints `[RUN STATUS] SUCCESS` only after successful completion.

Detailed events include `[START]`, `[ENV]`, `[FEATURE]`, `[CHRONOLOGY]`, `[DATA]`, `[MODEL]`, `[SUPPORT]` (manifests), `[EPISODE]`, `[ADAPT MATCHING]`, `[SELECT]`, `[FACULTY RESULTS]`, `[MACRO RESULTS]`, `[COVERAGE]`, `[REPORT]`, `[FAILURE]` and `[RUN STATUS]`. The official fine-tuning training loop also prints its original epoch losses and validation progression with `verbose=True`.

Per-output directory:

| Artifact | Purpose |
|---|---|
| `config.json` | Full resolved launch arguments |
| `environment.json` | Python, sklearn, Torch, TabICL/TabPFN versions and GPU |
| `feature_contract.json` | Actual ordered features, dimension and schema hash |
| `all_support_manifests.jsonl` | Exact support row identities, origins, budget, seed, prevalence, hash |
| `all_resource_profiles.csv` | Query resource profile per faculty/context |
| `fold_scores/F*_*.csv` | AP/AUC/F2/Brier/ECE + other metrics per faculty–origin |
| `predictions/F*_*.csv` | Exact row keys and probabilities; potentially sensitive |
| `checkpoints/F*_*/best.ckpt` | Upstream-compatible TabICLv2 fine-tuning checkpoint |
| `checkpoints/F*_*/selection_curve.csv` | Historical macro AP, validation Brier, batch losses and updates |
| `all_metrics.csv` | All scored cell metrics |
| `all_macro_by_fold.csv` | Macro AP by fold, model and budget |
| `all_macro_by_model.csv` | Aggregated scorecard |
| `all_paired_deltas.csv` | Paired AP improvements and win/loss counts |
| `coverage_manifest.csv` | Completed and failed jobs |
| `frozen_development_protocol.json` | Protocol lock written on successful development stage |

Reprint report without training:

```bash
bash run_suite31.sh --stage report --output-dir "$SUITE31_ROOT/results/suite31_CAPT_pilot"
```

Replay a saved prediction configuration using exact feature/cache keys and deterministic support selection:

```bash
bash run_suite31.sh --stage replay --device cuda --faculties all \
  --scores "$SUITE31_ROOT/results/suite31_CAPT_pilot/predictions/F1_tabicl_frozen_B4096_S29.csv" \
  --output-dir "$SUITE31_ROOT/results/suite31_CAPT_pilot"
```

For adapted-score replay, also pass `--checkpoint <best.ckpt>`. Replay fails if per-row probabilities differ by more than `1e-4`. Use exactly the original software versions and settings. It may have numeric sensitivity across GPU library versions.

**Data privacy:** row-level scores contain real student identifiers. Do not share those outputs unredacted.

## 7. Honest limitations and scientific safeguards

1. The code preserves the existing **semester-level** cache and cannot certify actual upstream grade-entry/release timestamps. `--release-gap` provides a coarse embargo only.
2. The original raw cache contains NaNs zero-filled by the historical loader. No new faculty-native typing or native feature recovery is attempted in Suite31.
3. `TabICLv2` and `TabPFN` require their real model packages/weights. Neither is shipped in the ZIP; the initial local test environment cannot validate their GPU integration.
4. The adaptation branch uses a source-level hook for TabICL's private `_finetune.base.iter_epoch_meta_batches`. It **fails on incompatible internals**. Pin the version you validate.
5. The framework compares faculty-local supports and does not add cross-faculty labeled contexts in the initial study.
6. Some historical validation groups may have no valid two-class support; these use the declared fallback. Single-class evaluation AP/AUC are undefined.
7. Primary metric: equally weighted eligible faculty–origin AP. Never interpret this as pooled AP or student-level warning quality.
8. The V100 may not support the proposed 16,384 context size with the requested ensemble; record feasibility and do not silently replace a model's context.
9. The suite does **not** assert pretraining improves AQ; this is exactly the hypothesis to be tested.
10. Repeated F6/F7 consultation means only a genuinely later untouched semester can support a prospective research claim.
11. The frozen transfer controls intentionally compare different architectures; good transfer against matched HGB is informative but not a causal isolation of external pretraining alone.
12. Unit tests and synthetic HGB tests do not demonstrate TabICL fine-tuning success. **Run the actual V100 smoke paths and confirm checkpoint replay before investing in a full run.**

## 8. Sources

- Primary task/research brief `AQ_research_next_directions.md` (also included in `docs/`).
- Original Suite30 loader source retained in `legacy/` under the same function contract.
- TabICLv2: https://github.com/soda-inria/tabicl and https://arxiv.org/abs/2602.11139
- TabICLv2 fine-tuning internals: https://github.com/soda-inria/tabicl/blob/main/src/tabicl/_finetune/base.py and `data.py`
- TabPFN: https://github.com/PriorLabs/TabPFN (model weights independently licensed).
