# Suite30 — Joint Semester Portfolio Risk (JSPR)

## Research question
> What information about a student's simultaneous course portfolio genuinely
> helps predict individual and collective academic failure, beyond what is
> already captured by course features, student history, and cohort-level risk?

Suite30 is a **single integrated model** that estimates course failure
probabilities **and** a joint distribution over the semester's vector of
failure outcomes. It is not a post-hoc ensemble or a blend with the Committee.

**Scientific status:** Source-level and synthetic end-to-end validation have
passed. **The actual university caches have not been executed in the author's
environment.** A real-data smoke test is required before the full run.

## Compatibility and directory layout
The cache contract is inherited verbatim from Suite29 v3.1 / Suite25:
- Canonical: `results/cross_faculty_residual_moe/cache/<faculty>_fold_<n>.joblib`
- Course features: `results/suite12_structured_cache/<faculty>_fold_<n>.joblib`
- Raw feature intersection across selected faculties, ordinarily 136 columns
- 17 `course_target` structured features, ordinarily total **153**
- Groups `(faculty, IDSV, target_NHHK)`, unordered.
- Historical training: cross-fitted covered rows, subject to the group audit.
- Global calendar cutoffs for historical validation and target separation.

The generated archive contains a single top-level `suite30_joint_portfolio/`.
Extract it at `/root/disk_3tb/earlywarning/30072026/`.

```
/root/disk_3tb/earlywarning/30072026/
├── suite30_joint_portfolio/
│   ├── run_suite30.sh
│   ├── train_suite30.py
│   ├── suite30_model.py
│   ├── suite30_data.py
│   ├── suite30_utils.py
│   ├── suite30_report.py
│   ├── verify_checkpoint.py
│   ├── check_suite30_environment.py
│   ├── suite25_common.py, suite25_data.py, suite25_reporting.py
│   └── tests/
├── log/
│   └── suite30_joint_portfolio.log             # SINGLE stable log; overwritten per run
└── results/
    ├── suite29_enrollment_set/                 # optional exact-key verified references
    ├── suite30_smoke/                          # use separate smoke output
    └── suite30_joint_portfolio/                # full final experiment
```

The launcher runs preflight, unit tests and training, saving terminal output to
exactly **one log file**. `tee` is invoked **without `-a`**, so every execution
replaces the prior run's log. Copy it before rerunning if you need the old log.

## Commands

### Install / validate
```bash
unzip suite30_joint_portfolio.zip -d /root/disk_3tb/earlywarning/30072026/
cd /root/disk_3tb/earlywarning/30072026/suite30_joint_portfolio
/root/miniconda3/envs/Huy/bin/python3 -m unittest discover -s tests -v
```

### Recommended REAL-CACHE all-faculty smoke
A single-faculty run changes the common raw feature intersection, so a
one-fold **all-faculty** check is more informative.
```bash
bash run_suite30.sh \
  --folds 1 --faculties all --seeds 29 \
  --epochs 2 --min-epochs 1 --patience 1 \
  --group-batch-size 32 --bootstrap 0 \
  --output-dir /root/disk_3tb/earlywarning/30072026/results/suite30_smoke
```
This is a **compatibility smoke test**, NOT an estimate of model quality.

### Full experiment
```bash
cd /root/disk_3tb/earlywarning/30072026/suite30_joint_portfolio
bash run_suite30.sh
```
Defaults: **all 7 folds × 14 faculties × 3 seeds (29,129,229)**; four
separately trained model variants; 18-epoch budget, early stopping with
patience=5, width=128, heads=4, 64 groups/batch, checkpoints, bootstrap=100.
This involves **84 separately initialized model fits for selection, each
followed by one full-history refit**. It is more expensive than Suite29.

Monitor:
```bash
tail -f /root/disk_3tb/earlywarning/30072026/log/suite30_joint_portfolio.log
```

`PYTHON=/custom/python` and `AQ_ROOT=/custom/root` can override the defaults.
CLI arguments passed after `bash run_suite30.sh` override the launcher defaults.

## Architecture: one coherent joint distribution
Within a group of M real concurrently enrolled courses, the shared course
encoder maps each `[153]` registration to a width-128 token.
The peer branch uses masked, **leave-one-out** 4-head attention on the 17
target-course feature channels (queries use the whole course token). Padding
and the querying course itself are excluded from peer keys.

The shared course representation and peer message contribute to a base logit:
```
r_j = row_head(x_j) + sigmoid(peer_gate_j) * peer_effect_j
```
A separate portfolio encoder pools the group representations, course feature
representations, course-feature variance and log set size. It parameterizes a
two-component latent Bernoulli risk state:
```
pi    = sigmoid(group_prior)
p0_j  = sigmoid(r_j - softplus(low_shift))
p1_j  = sigmoid(r_j + softplus(high_shift))
p_j   = (1-pi) * p0_j + pi * p1_j

P(y_1,...,y_M | X)
      = (1-pi) * product_j Bern(y_j;p0_j)
        + pi * product_j Bern(y_j;p1_j)
```
All components are in **one neural model trained end-to-end**.
The model evaluates exact Poisson-binomial component count PMFs (a mixture of
two component PMFs), yielding expected failures, P(any failure), P(>=2 failures),
and a properly normalized count distribution. The shared latent state permits
positive within-semester failure covariance.

### Training modes (separate fits)
1. `course_only`: no peer message, independent per-course outcomes.
2. `portfolio_independent`: peer attention, independent outcomes.
3. `joint_no_attention`: shared latent joint risk, portfolio pooling, no
   direct leave-one-out attention. IMPORTANT: it **still** uses aggregated
   group context; the name only excludes attention.
4. `joint`: full peer attention + learned joint risk state (primary model).

All variants use the same raw/structured features, chronological splits, seed
configuration, optimizer family, and common architecture definition; the
enabled computations and trainable pathways differ intentionally.

### Interventions on the fitted joint network
- `suite30_joint_no_attention_intervention`: block the direct attention branch,
  retaining group-level risk pooling.
- `suite30_joint_no_context_intervention`: remove direct attention and group
  mixture, yielding the row-only pathway of that SAME fitted joint network.
- `suite30_joint_matched_placebo`: substitute each group's COURSE FEATURE
  tokens from a different student group with EXACT same faculty, semester
  ordinal and group size. Nonmatched groups remain unchanged and matching
  coverage is reported; matched-only AP deltas are reported separately.
  The subject's primary row representations remain authentic, so this is a
  **matched peer-course-substitution experiment**, not a complete-group swap.

This comparison distinguishes authentic portfolio composition from generic
group size or having access to a peer key. It does not establish causal effects
without additional assumptions.

## Objective, curriculum and sampling
Each step computes:
- Registration-weighted marginal BCE
- Registration-normalized negative log likelihood for the joint outcome vector
- Group-level BCE for any failure and multiple failures
- Smooth L1 for observed failure *rate* versus expected failure rate per group
- Bounded hard-negative pair-ranking loss, added as a regularizer

Default coefficients are:
```
marginal BCE:       1.00
joint vector NLL:   0.20  (joint modes only)
any-failure BCE:    0.03  (joint modes only)
multiple BCE:       0.03  (joint modes only)
failure-rate L1:    0.03  (joint modes only)
ranking loss:       0.05
```
The additional terms ramp up during the first **two** epochs (starting in
epoch 1). This is intentional: Suite29 often selected epochs 2–5, so a
two-epoch zero-loss warm-up could prevent the new joint head from ever
learning. Change `--warmup-epochs` and `--ramp-epochs` to test schedules.

The historical train loader uses faculty-aware **replacement sampling** with
power 0.5 by default. Printed `train_rows` means **row exposures within that
epoch**, not the number of unique registrations seen, which may differ.
A six-semester recency half-life with a 0.35 weight floor is applied to
training objectives. All weights depend only on historical semester metadata.

### Probability calibration without losing coherence
At the *selected historical validation epoch*, fit a global intercept and a
shrunken per-faculty intercept by minimizing historical validation BCE.
Do **not** use target-fold outcomes. Apply the same intercept shift to BOTH
component logits (`p0_j`, `p1_j`) before mixing and recomputing the full
count PMF. Thus all reported course/collective probabilities remain consistent.

The fitted intercepts are stored in every checkpoint. The uncalibrated primary
joint prediction is also saved as `suite30_joint_raw_uncalibrated`. This is
historical **intercept** recalibration, not a guarantee against sudden changes
in future target prevalence. It uses the same historical validation period as
early stopping, so its in-validation calibration diagnostics are descriptive.

Disable with `--calibration none`, disable recency weighting with
`--recency-half-life 0`, or faculty sampling with
`--faculty-sampling-power 0`.

## Evaluation and chronological integrity
1. Discover raw common features from the selected faculties; ensure all future
   folds provide this schema.
2. Load each fold's historical *cross-fitting-covered* rows and untouched
   target registrations.
3. Validate no historical semester is at or after the target period.
4. Select training epochs on the latest 3 historical calendar semesters, with
   one **global cutoff shared across ALL faculties**. Validation targets do
   not update training weights.
5. Fit historical-only intercept calibration at the selected best epoch.
6. Reinitialize with the same seed and refit for that many epochs on all
   eligible history. Predict the target semester only after refitting.
7. Apply the historical intercept and mechanistic interventions. Save the
   original row order and exact registration keys.
8. Compute per-faculty AP, AUC, precision/recall/F2 at 10/15/20/25% warning
   budgets, Brier, ECE, plus any/multi/count errors and joint NLL.

**Primary scientific questions:** Does full joint outperform
`portfolio_independent` and `joint_no_attention`? Does its matched-only
portfolio AP exceed the placebo? Is F6/F7 gain persistent? Is count/multiple
failure calibration materially better? Does it beat archived Suite29 Set and
Committee at the SAME fully covered faculty/fold cells?

`F6/F7` have been consulted repeatedly in earlier experiments: **retrospective
only**, never an untouched future test set. Do not select another architecture
using the retrospectively inspected outcomes and then claim novel holdout AP.

## Exact archived reference alignment
If old Suite29 outputs exist at
`results/suite29_enrollment_set/fold_<fold>_seed_<seed>_scores.csv`, Suite30
automatically loads `suite29_self_only`, `suite29_set`, `suite29_set_no_peers`.
It **validates every faculty, student ID, course ID, semester, row index, and
actual outcome in exact row order** before scoring. Mismatch raises an error;
missing CSV emits an explicit warning. Disable with `--suite29-results ""`.

For Committee/Suite29 (which do not output a trained joint distribution),
group-level probability/count diagnostics use the product-of-independent-
Bernoullis approximation from their marginal scores. Comparisons of joint
NLL must disclose this distinction.

The archived Committee probabilities are included from the same canonical
target cache, NEVER used as inputs or blended into new predictions.

## Main outputs
- `fold_<fold>_seed_<seed>_scores.csv` — original row keys and target labels;
  all available score columns; peer gate/entropy/top weight; joint-state
  separation and P(any/multiple); matched-only row flag.
- `fold<fold>_<mode>_seed<seed>.pt` — checkpoint, architecture config,
  preprocessing means/scales, historical calibration intercepts, feature order
  and faculty order.
- `all_metrics.csv`, `all_course_size_metrics.csv` — per-faculty AP/AUC/F2/Brier
  and course-set-size strata across all models.
- `all_group_diagnostics.csv` — student–semester count, any/multiple failure,
  coherent joint NLL, peer effect and positive-dependence excess variance.
- `all_joint_group_summary.csv`, `all_selected_epochs.csv`,
  `all_selection_curves.csv`, `all_selection_faculty_curves.csv`,
  `all_refit_losses.csv`, `all_portfolio_diagnostics.csv`.
- `all_historical_calibration.csv` — calibration bias and validation BCE
  (no target labels used in fitting).
- `all_matched_peer_deltas.csv`, `fold_*_placebo_coverage.csv`.
- `all_history_coverage_audit.csv`, `all_bootstrap.csv`,
  `all_target_enrollment_profiles.csv`.
- `summary_*.csv` — macro across folds/seeds, retrospective F6/F7, all paired
  comparisons, faculty stability, faculty × group size, temporal calibration,
  overfitting, matched portfolio placebo, coverage and other diagnostics.
- `config.json`, `features.json`, `runtimes.csv`.

To **rerender the report without retraining**:
```bash
/root/miniconda3/envs/Huy/bin/python3 suite30_report.py \
 --output-dir /root/disk_3tb/earlywarning/30072026/results/suite30_joint_portfolio
```

To **independently replay a checkpoint**:
```bash
/root/miniconda3/envs/Huy/bin/python3 verify_checkpoint.py \
 --checkpoint /root/disk_3tb/earlywarning/30072026/results/suite30_joint_portfolio/fold1_joint_seed29.pt \
 --scores /root/disk_3tb/earlywarning/30072026/results/suite30_joint_portfolio/fold_1_seed_29_scores.csv
```
Only load checkpoints you trust (`torch.load` uses object deserialization).

## Reproducibility and limitations
- 19 unit tests (grouping, masking, permutation equivariance, gradients,
  probability consistency, joint covariance, exact placebo matching,
  chronology, historical calibration, strict reference identities).
- Executed two-faculty synthetic tests across 2 folds and all 4 train modes;
  full synthesis includes placebo, calibration, save/replay and final reports.
- All test AP values are from **synthetic caches**; they say nothing about
  expected real university performance.
- The upstream provenance of all engineered historical features has NOT been
  audited to confirm availability at the prediction cutoff. Bad upstream
  feature timing cannot be fixed by a downstream attention mask.
- Grouping is **within faculty**, not necessarily a student's full university
  portfolio. Inter-faculty registrations may be absent.
- The historical cross-fit mask can exclude entire historical enrollment sets.
  Partial retained groups are printed and optionally rejected.
- The latent two-state family captures **positive shared risk**, not arbitrary
  high-order correlations or negative dependence.
- An improvement after placebo comparison supports nontrivial peer relevance
  but does not identify causally responsible courses.
- Calibration may still deteriorate sharply under the target prevalence drift
  seen in Suite29. Explicitly evaluate it rather than treating it as solved.
- The experiment is not guaranteed to outperform Suite29 or the Committee.
  Its purpose is a falsifiable and substantially redesigned scientific test.
