# Leak-Safe Feature Experiment Suite Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build a reproducible E0–E9 experiment suite that preserves the effective-number MultiTaskNet baseline and adds staged, strictly prior-semester feature cleaning and historical feature engineering.

**Architecture:** Raw expanding-fold CSVs are loaded without using test labels for feature construction. A semester-ordered historical state emits student, course, and student-course features before updating with that semester's outcomes. Feature-set manifests compose cumulative E0–E9 experiments, and one shared trainer/evaluator guarantees identical modeling and metrics across experiments.

**Tech Stack:** Python 3.10+, NumPy, pandas, PyTorch, pytest; optional tqdm.

## Global Constraints

- Positive event is `target_grade10 < 5.0`.
- Preserve original MultiTaskNet architecture: trunk `(256, 128)`, head `64`, BatchNorm, ReLU, dropout `0.3`.
- Preserve AdamW `lr=1e-3`, weight decay `1e-5`, low-grade MSE weight `5.0`, effective-number BCE with beta `0.9999`, and three equal loss coefficients.
- Imputation and z-normalization statistics must come from the current fold's training data only.
- Raw `IDSV`, `IDMH`, `IDKQDK`, and absolute semester identifiers are metadata, never neural-network inputs.
- Historical features for a training row may use only semesters strictly earlier than its target semester.
- Test feature construction may use all completed training semesters but must never read test grades.
- E0 must preserve the original baseline feature selection and preprocessing behavior.
- Folds 1–5 are development; folds 6–7 are exploratory holdout confirmation.

---

### Task 1: Configuration, fold loading, and baseline compatibility

**Files:**
- Create: `early_warning_suite/config.py`
- Create: `early_warning_suite/data.py`
- Test: `tests/test_data.py`

**Interfaces:**
- Produces `SuiteConfig`, `FoldData`, `load_raw_fold()`, `baseline_feature_columns()`.

- [ ] Write tests for target filtering, feature alignment, and E0 feature selection.
- [ ] Run tests and confirm imports fail.
- [ ] Implement configuration and fold loading.
- [ ] Run tests and confirm pass.

### Task 2: Leak-safe historical state

**Files:**
- Create: `early_warning_suite/historical.py`
- Test: `tests/test_historical.py`

**Interfaces:**
- Produces `build_leak_safe_historical_features(train_df, test_df, config)`.

- [ ] Write tests proving same-semester and future-label invariance.
- [ ] Run tests and confirm imports fail.
- [ ] Implement student, course, and student-course historical aggregates.
- [ ] Run tests and confirm pass.

### Task 3: Cleaning, transformations, and experiment manifests

**Files:**
- Create: `early_warning_suite/feature_sets.py`
- Test: `tests/test_feature_sets.py`

**Interfaces:**
- Produces `ExperimentSpec`, `EXPERIMENTS`, `prepare_experiment_matrices()`.

- [ ] Write tests for E0–E9 cumulative behavior, train-only constant removal, course-status ablation, redundancy pruning, and finite aligned matrices.
- [ ] Run tests and confirm imports fail.
- [ ] Implement staged feature preparation.
- [ ] Run tests and confirm pass.

### Task 4: Baseline-compatible model training

**Files:**
- Create: `early_warning_suite/model.py`
- Test: `tests/test_model.py`

**Interfaces:**
- Produces `MultiTaskNet`, `effective_number_weights()`, `train_predict()`.

- [ ] Write architecture and class-weight tests.
- [ ] Run tests and confirm imports fail.
- [ ] Implement the original model and losses.
- [ ] Run tests and confirm pass.

### Task 5: Metrics, prediction records, and reports

**Files:**
- Create: `early_warning_suite/metrics.py`
- Test: `tests/test_metrics.py`

**Interfaces:**
- Produces AP/AUC, threshold tuning, confusion metrics, equal-warning-budget metrics, and pooled reporting helpers.

- [ ] Write deterministic metric tests.
- [ ] Run tests and confirm imports fail.
- [ ] Implement metrics.
- [ ] Run tests and confirm pass.

### Task 6: Experiment runner and output artifacts

**Files:**
- Create: `early_warning_suite/runner.py`
- Create: `run_experiments.py`
- Test: `tests/test_runner.py`

**Interfaces:**
- Produces CLI execution for one/all experiments and CSV/JSON outputs.

- [ ] Write smoke tests with synthetic folds and one training epoch.
- [ ] Run tests and confirm imports fail.
- [ ] Implement runner and comparison table.
- [ ] Run tests and confirm pass.

### Task 7: Error analysis

**Files:**
- Create: `early_warning_suite/error_analysis.py`
- Create: `inspect_predictions.py`
- Test: `tests/test_error_analysis.py`

**Interfaces:**
- Produces TP/FN/FP/TN subgroup tables from saved predictions.

- [ ] Write subgroup-analysis tests.
- [ ] Run tests and confirm imports fail.
- [ ] Implement error analysis and CLI.
- [ ] Run tests and confirm pass.

### Task 8: Documentation and end-to-end verification

**Files:**
- Create: `README.md`
- Create: `requirements.txt`
- Create: `early_warning_suite/__init__.py`

- [ ] Document experiment meanings, commands, leakage guarantees, and interpretation.
- [ ] Run `python -m compileall`.
- [ ] Run full pytest suite.
- [ ] Run feature-generation smoke test on actual IT fold data.
- [ ] Run one-epoch E0/E9 model smoke tests.
- [ ] Package source and tests into a ZIP archive.
