# Suite2 Direct Improvement Experiments Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build four controlled chronological experiments that directly test strong nonlinear meta-learning, relational fourth-expert stacking, history-depth regime specialization, and multi-window temporal ensembling against the unchanged V2 tree committee.

**Architecture:** `suite2_common.py` reuses `train_v3.py`/`breakthrough_common.py` for fold loading, RF/ExtraTrees/HistGB configuration, chronological cross-fitting, metrics, and relational primitives. Each experiment has an isolated CLI/output directory and writes fold-level predictions plus diagnostic CSVs. `run_suite2.sh` launches all four concurrently without sharing mutable state.

**Tech Stack:** Python 3, NumPy, pandas, scikit-learn, PyTorch (relational expert only), existing `train_v3.py` and breakthrough utilities.

## Global Constraints

- Target is `target_grade10 < 5`.
- Development folds are 1–5 and holdout folds are 6–7 by default.
- Holdout labels are evaluation-only; they never tune specialist, stacker, route, blend, regime, or window selection.
- Base RF/ExtraTrees/HistGradientBoosting definitions must match `train_v3.py`.
- Experiments must expose committee counterfactuals and fold-level AP/AUC/F2 diagnostics.
- Every experiment writes to its own directory under `suite2` output root.
- GPU use is optional and only applies to PyTorch relational training; sklearn tree models remain CPU.

---

### Task 1: Shared chronology, meta features, support features, and HGB selection

**Files:**
- Create: `suite2/suite2_common.py`
- Test: `suite2/tests/test_suite2.py`

**Interfaces:**
- Produces `v2_meta_features`, `chronological_support_features`, `fit_hgb_candidate`, `select_hgb_with_authority`, `equal_budget_metrics`, `pooled_summary`.
- Consumes canonical base committee/data functions from `train_v3.py` and `breakthrough_common.py`.

- [ ] Write failing tests for V2 meta dimensionality, chronology-safe history depth/course novelty, authority-0 identity, and HGB probability shape.
- [ ] Run `pytest -q suite2/tests/test_suite2.py` and confirm missing-module failure.
- [ ] Implement the shared module.
- [ ] Run focused tests and confirm green.

### Task 2: Experiment A — Strong Tree Meta/Specialist

**Files:**
- Create: `suite2/exp_a_tree_specialist.py`
- Test: `suite2/tests/test_suite2.py`

**Interfaces:**
- Produces two candidates per fold: `allrow_meta_hgb` and `routed_meta_hgb`.
- Uses full prepared features + V2 committee meta + support features.

- [ ] Add failing tests for candidate feature construction and routed fallback identity.
- [ ] Implement chronological OOF meta training, multi-block validation, HGB candidate/authority selection, final refit, target prediction, and diagnostics.
- [ ] Save fold scores, candidate grid, route diagnostics, feature importance, subgroup/support summaries, and pooled summary.
- [ ] Verify unit tests.

### Task 3: Experiment B — Relational Fourth Expert Stacking

**Files:**
- Create: `suite2/exp_b_relational_stacker.py`
- Test: `suite2/tests/test_suite2.py`

**Interfaces:**
- Produces chronological hybrid-relational OOF probabilities, then `lr_stack4` and `hgb_stack4` predictions.
- Uses novelty/history support as competence features.

- [ ] Add failing tests for OOV-safe relational encoding and four-expert stack meta dimensions.
- [ ] Implement chronological relational cross-fitting and production relational refit.
- [ ] Implement original-V2-style logistic stacker and nonlinear HGB stacker, both selected without holdout tuning.
- [ ] Save relational block diagnostics, stacker validation metrics, coefficients/importances, cold-start subgroup results, fold scores, and pooled summary.
- [ ] Verify unit tests.

### Task 4: Experiment C — History-Depth Regime Models

**Files:**
- Create: `suite2/exp_c_history_regime.py`
- Test: `suite2/tests/test_suite2.py`

**Interfaces:**
- Regimes: `history0`, `history1_2`, `history3plus` based on strictly prior semesters.
- `history3plus` can use committee + hybrid relational expert; limited/no-history regimes use HGB meta models with regime-appropriate features and committee fallback.

- [ ] Add failing tests for deterministic regime labels.
- [ ] Implement chronology-safe depth computation for historical and target rows.
- [ ] Implement per-regime temporal validation/authority selection and production refit.
- [ ] Save regime sizes, failure rates, selected models/authority, per-regime AP/AUC/F2, feature importance, fold scores, and pooled summary.
- [ ] Verify unit tests.

### Task 5: Experiment D — Multi-Window Temporal Specialist Ensemble

**Files:**
- Create: `suite2/exp_d_multiwindow.py`
- Test: `suite2/tests/test_suite2.py`

**Interfaces:**
- Windows: all history, last 6, last 4, last 2 supported temporal blocks.
- Uses prequential block predictions to select `committee`, `equal_window_mean`, or regularized logistic window stacker; committee fallback is always eligible.

- [ ] Add failing tests for window index construction and probability-preserving fallback.
- [ ] Implement prequential predictions for each temporal window.
- [ ] Fit/select robust temporal blend on historical transitions only; refit selected window models on all eligible history.
- [ ] Save transition metrics, window contributions, stacker coefficients, fold scores, and pooled summary.
- [ ] Verify unit tests.

### Task 6: Launcher, README, synthetic integration, and packaging

**Files:**
- Create: `suite2/run_suite2.sh`
- Create: `suite2/README_suite2.txt`
- Create: `suite2/make_synthetic_suite2_data.py`
- Create: `suite2/suite2_bundle.zip`

**Interfaces:**
- Launcher accepts split directory, output root, and faculty; `DEVICE=auto|cpu` controls PyTorch experiments only.

- [ ] Add shell syntax verification.
- [ ] Generate small chronological synthetic folds with unseen students/courses in later semesters.
- [ ] Run each experiment independently with reduced smoke-test hyperparameters.
- [ ] Verify expected summaries/score CSVs exist and contain finite committee/candidate probabilities.
- [ ] Run full `pytest`, `py_compile`, shell syntax check, and ZIP integrity check.
