# Suite30 JSPR — Design audit, hypothesis tests and implementation contracts

## Premise and scientific estimand
Investigate whether the *composition* of a student's simultaneous
faculty-observed course portfolio adds genuine predictive information beyond
existing student history, cohort/course features and the set size.

Primary contrast (paired faculty × fold × seed macro AP):
`joint` versus `portfolio_independent`, `joint_no_attention`,
and the archived fully aligned Suite29 Set + Committee.

For true portfolio-information attribution:
- `joint` versus fitted `joint_matched_placebo` on **matched-only** target rows.
- If a faculty-semester-set-size stratum has no other student donor, leave its
  predictions unaltered and report exact matched fractions.
- No outcome label is consulted for donor selection; no target labels are fed
  to the network or to historical calibration.

Collective target:
- Joint Bernoulli-vector log likelihood (not a product of marginal outputs).
- Group P(any), P(multiple), expected count and count variance/correlation.
- Course-level AP/F2/calibration and faculty × set-size breakdowns.

## Model mathematical invariants
Let p_j^0, p_j^1 be conditional failure probabilities given a binary shared
semester state S and w=P(S=1). Then:
```
p_j=(1-w)*p_j^0+w*p_j^1
P(Y=y)=(1-w)*prod Bern(y_j;p_j^0)+w*prod Bern(y_j;p_j^1)
P(K=k)=(1-w)*PoissonBinomial_k(p^0)+w*PoissonBinomial_k(p^1)
E[K]=sum_j p_j
Cov(Y_j,Y_l)=w*(1-w)*(p_j^1-p_j^0)*(p_l^1-p_l^0), j!=l
```
`softplus` enforces p_j^1 >= p_j^0. Therefore pairwise covariance is
nonnegative. This is a constrained family and cannot capture arbitrary
dependencies: report rather than conceal that limitation.

Historical-only calibration is a **shared logit shift** applied to both
conditional states. A post-hoc adjustment to p_j without recomputing the
conditional states and count PMF is forbidden.

## Inputs/chronology
- Features: common canonical raw schema + exact 17 course-specific historical
  features. Group by faculty, student ID and target semester.
- The global train/validation split uses *one common earliest validation term*
  across faculties. It fails if train and validation share one group.
- The target fold is strictly after all historical labels.
- Standardization (mean, scale, semester mean/scale) uses historical training
  only in selection, and eligible history in refit.
- Epoch selection uses only historical validation AP. Refitting starts with a
  fresh model from the same seed; target labels never select epochs.
- Historical-only calibration uses the selected historical checkpoint and
  its validation predictions; it does NOT use target-fold outcomes.
- Faculty-aware group sampling is replacement sampling: epoch exposures may
  repeat registrations. All sampler settings saved in config/checkpoints.

## Controls and semantic distinctions
- course_only: no peer attention and no group latent-risk mixture.
- portfolio_independent: peer attention without group latent-risk mixture.
- joint_no_attention: group latent-risk mixture (uses aggregate group
  features), no direct leave-one-out attention.
- joint: group latent-risk mixture with peer attention.
- joint_no_attention_intervention: mask direct attention on fitted joint model.
- joint_no_context_intervention: remove direct peer and shared mixture
  modulation on fitted joint weights. This is distribution shift, a diagnostic
  intervention, not a separately trained matching baseline.
- joint_matched_placebo: inject donor course-feature tokens; keep anchor row
  features untouched. This does not replace all shared group state.

Attention mass to peers is naturally approximately one whenever a query has
at least one valid peer: **do not cite that as evidence of meaningful peer
learning**. Report attention top weight, normalized entropy, gate and actual
absolute peer-induced logit shift. Matched placebo is the stronger test.

## Audit checkpoints
1. Unit tests: zero singleton peer message; permutation equivariance;
   padding invariance; normalized count PMF; E[K] identity; positive covariance;
   gradient through latent/peer branches; valid group IDs; exact reference keys.
2. Historical cache and row coverage printout per faculty/fold; optionally
   `--strict-complete-history-groups`.
3. Selection: chronology, per-faculty coverage, losses, AP/F2/Brier,
   any/multi validation calibration and best epoch.
4. Intercept calibration: global/per-faculty shift, shrinkage and historical
   validation BCE before/after.
5. Refit: one initialization with same seed and selected epochs, weighted
   loss components, epoch row exposures.
6. Targets: one score per registration, no duplicate assignment;
   group-level coherent probabilities and unconditional Committee reference.
7. Suite29 reference exact keys: faculty, student ID, course ID, semester,
   original index, actual binary outcome.
8. Placebo exact match counts, overall intervention AP, matched-only AP.
9. Faculty × fold × seed paired performance; set-size strata and group
   count/multiple-failure calibration, latent risk and peer effectiveness.
10. Checkpoint replay of target predictions and all primary interventions
    to <=1e-6 absolute deviation.

## Synthetic verification carried out
- Python model structural tests, chronology, data grouping, placebo, strict
  external-row identity, binary count overflow regression.
- Full synthetic-cache load from canonical and structured joblib formats.
- Chronological selection/refit for all four modes, at 2 epochs.
- Count/any/multiple group metrics, all CSV summaries, calibration diagnostics,
  paired comparison, target score CSV and save-checkpoints.
- `verify_checkpoint.py` successfully reproduced primary and intervened
  predictions on synthetic cache rows to floating-point tolerance.
- Two-fold synthetic report covers all summary generation paths.

These are SOFTWARE integration tests, not performance evidence.

## Known limitations
- Full 14-faculty university caches were not locally available. Do not infer
  expected AP from the synthetic integration test.
- Upstream engineered feature timing remains unaudited. A feature derived from
  outcomes after the prediction cutoff would contaminate the experiment even
  if Suite30's own chronology and masks are correct.
- Faculty-scoped groups may not include other faculties' enrollments.
- Historic CF mask may delete whole groups; no partial groups is not equal to
  full population coverage.
- F6/F7 are repeatedly inspected retrospective targets, not independent
  untouched test folds. Future confirmation must use new time periods.
- The two-state mixture can learn positive dependence but not negative or
  unrestricted higher-order dependence.
- Within a fixed set size, attention top-weight and entropy are descriptive,
  not causal proof; matched placebo is stronger but also not causal.
- Historical validation calibration could overfit the same window used for
  early stopping. This is disclosed; target performance must be checked.
- Row-level comparison with older suites must verify exact keys; missing or
  unverified baseline rows are not filled silently.
