# Suite21 design specification

## Name

**Suite21 — Committee-Anchored Guarded Temporal Residual**

## Frozen research question

Suite20 established that ordered semester history contains useful predictive information, but its weak neural current-state branch and aggressive temporal gate caused calibration error and false negatives. Suite21 therefore asks a narrower final question:

> Can explicit semester history improve the already-strong Committee if temporal evidence is allowed to act only as a bounded, validation-controlled residual correction?

This is intentionally the last experiment in the current temporal-Transformer family unless it produces clear gains.

## Core principle

The predictor is no longer learned from scratch. For each sample,

`base_logit = logit(Committee probability)`

and Suite21 only proposes

`final_logit = base_logit + alpha * bounded_residual`.

`alpha=0` is included in every validation authority grid and exactly recovers the Committee. Therefore the temporal model must earn authority on chronological validation data.

## Why the Committee is a safe anchor

Historical rows use the existing chronology-safe cross-fitted Committee probabilities (`cf_probabilities`). Target rows use production Committee expert probabilities (`pte`). Thus the residual learner never sees in-sample Committee scores for historical fitting.

## Three ablations

1. `flat_residual`
   - Committee anchor + current flat/tabular context.
   - No explicit student semester history.
   - Tests whether a generic neural residual alone can improve the Committee.

2. `temporal_residual`
   - Same Committee anchor.
   - Adds target-conditioned ordered-semester Transformer.
   - Symmetric bounded residual.
   - Isolates temporal complementarity over the strong anchor.

3. `guarded_temporal_residual` **(primary)**
   - Same temporal architecture.
   - Negative/downward corrections are increasingly restricted as Committee risk rises.
   - Directly addresses Suite20's false-negative pattern where historical stability could suppress acute failure risk.

## Guarded residual

The shared residual ensemble predicts a signed proposal `r in [-1,1]`.

Positive correction:

`delta_up = positive_cap * relu(r)`

Negative correction:

`delta_down = negative_cap * relu(-r) * (1 - q_committee)^guard_power`

The default caps are:

- `positive_cap = 1.50` logits
- `negative_cap = 1.00` logits
- `guard_power = 1.50`

This is deliberately asymmetric: historical evidence can rescue underestimated failure risk more freely than it can erase a strong Committee warning.

## Authority selection

After each epoch, the model is evaluated on the latest historical temporal blocks. A five-epoch minimum selection warm-up is used because the residual head starts at the exact Committee identity and needs several epochs before complementary corrections can emerge. A small deployment authority grid is tested without retraining:

`alpha = {0, .25, .50, .75, 1.0}`

Selection uses broad faculty-level improvement over Committee:

- mean AP delta,
- median AP delta,
- lower-quartile AP delta,
- mean F2@20 delta,
- a small penalty for severe worst-faculty regression.

Because `alpha=0` has exact delta 0, harmful residual models automatically fall back to Committee.

## Training changes from Suite20

Suite20's class/faculty effective-number resampling distorted the score prior and was associated with poor Brier score. Suite21 uses **faculty-balanced sampling only**. Class prevalence inside each faculty is preserved.

The final head is regularized with:

- primary BCE on Committee-anchored final logits,
- small squared residual penalty,
- small positive-preservation penalty discouraging downward corrections on known historical failures,
- low-weight temporal auxiliary BCE to keep the semester encoder informative,
- low-weight within-faculty hard-pair ranking loss focused on Committee mistakes (under-ranked failures vs over-ranked normal cases).

## Temporal architecture

The temporal encoder is the Suite20 semester Transformer:

- each prior semester is a chronological token,
- learned semester-position embeddings preserve order,
- target-course + faculty context forms a query,
- cross-attention retrieves target-relevant history.

Suite21 does **not** claim that target conditioning is independently proven; it preserves the strongest Suite20 temporal design while changing the fusion/deployment mechanism.

## Chronology and leakage rules

- Student history contains only semesters strictly earlier than the target semester.
- Historical Committee scores are OOF/cross-fitted.
- Latest historical blocks select epochs and residual authority.
- F6/F7 never select architecture, epoch, alpha, threshold, or warning policy.
- F1-F5 determine operational F2 threshold and warning ratio.
- F6/F7 remain final holdout.
- History reversal/zero-history experiments are post-hoc diagnostics only.

## Operational evaluation

Unlike Suite20's initial reporting, Suite21 automatically reconstructs the classic AQ operational protocol:

1. Pool F1-F5 for each faculty/model.
2. Select F2-optimal threshold.
3. Freeze threshold.
4. Compute development warning ratio = selected flag rate / development prevalence.
5. On each holdout fold, report both:
   - fixed-threshold decisions;
   - drift-aware warning rate based on the previous fold prevalence.
6. Print and save TP/FP/FN/TN confusion matrices.

This is applied to Suite21 and every earlier model for which aligned score files are present.

## Success criteria

Primary endpoint: F6/F7 14-faculty mean-fold macro AP.

Suite21 should be considered worthwhile only if it does more than preserve the Committee:

- `>= +0.005` macro AP with broad faculty support: meaningful.
- `>= +0.010` macro AP with no broad F2/recall regression: strong.
- Improvements concentrated in only one faculty or one holdout fold: hypothesis evidence, not a new project SOTA.
- Validation selecting `alpha=0` on many folds is an informative negative result: temporal corrections do not generalize reliably enough to earn authority.
