# Experiment B Feature Engineering Suite Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking.

**Goal:** Build a separate, reproducible experiment suite that keeps the accepted E0 neural model frozen and evaluates data cleaning, temporal stability selection, redundancy pruning, sparse-history representation, robust transformations, curriculum-status compression, and chronological cross-fitted risk bins.

**Architecture:** Reuse the verified Track A data loader, model, metrics, policy tuning, output schema, and error analysis. Replace Track A's cumulative feature-appending ladder with independent Experiment B feature pipelines. Every fold fits feature selectors and preprocessors using only that fold's training rows; B8 training-row risk encodings use labels only from earlier training semesters.

**Tech Stack:** Python 3, NumPy, pandas, PyTorch, tqdm, pytest.

## Global Constraints

- `B0` must reproduce the exact E0 baseline feature set, preprocessing, architecture, loss, optimizer, seed, and target filtering.
- `IDSV`, `IDMH`, target identifiers, targets, and target-semester metadata remain excluded from network inputs.
- No raw-table reconstruction is used; only the existing fold CSVs are consumed.
- Fold-test labels must never influence feature selection, correlation pruning, preprocessing, risk-bin edges, or risk estimates.
- B8 training rows in semester `s` may use labels only from training semesters earlier than `s`; the first training semester uses the fixed prior `0.05`.
- Folds 1–5 are development folds. Folds 6–7 are reused diagnostics, not pristine final holdouts.
- The package must use a separate output directory and must not overwrite Track A or E0→E9 outputs.

---

### Task 1: Baseline infrastructure and experiment registry

**Files:**
- Create: `experiment_b_suite/config.py`
- Create: `experiment_b_suite/experiments.py`
- Create: `experiment_b_suite/data.py`
- Create: `experiment_b_suite/model.py`
- Create: `experiment_b_suite/metrics.py`
- Test: `tests/test_registry_and_baseline.py`

**Interfaces:**
- `EXPERIMENTS: dict[str, ExperimentSpec]`
- `baseline_feature_columns(train_df, config) -> list[str]`
- Frozen `MultiTaskNet` and `train_predict(...)` copied without architectural changes from Track A.

- [x] Write a failing test asserting that B0, B1, B2_CS, B2_TIME, B2_CLUSTER, B2_REDUNDANT, B3, B4, B5, B6_W, B6_Q, B7, and B8 are registered.
- [x] Run the test and confirm failure because `experiment_b_suite` does not exist.
- [x] Implement the package configuration, registry, data loader, frozen model, and metrics.
- [x] Run the test and confirm it passes.

### Task 2: Structural and isolated cleaning pipelines

**Files:**
- Create: `experiment_b_suite/cleaning.py`
- Test: `tests/test_cleaning.py`

**Interfaces:**
- `structural_clean(train, test, config) -> FeatureTransformResult`
- `drop_course_status(train, test) -> FeatureTransformResult`
- `drop_time_features(train, test, config) -> FeatureTransformResult`
- `drop_cluster_identifier(train, test) -> FeatureTransformResult`

- [x] Write failing tests proving that structural cleaning removes train-all-null, train-constant, exact duplicate, unstable registration-timing, and absolute-semester columns while preserving ordinary E0 features.
- [x] Run tests and verify the expected missing-function failures.
- [x] Implement train-only structural cleaning and isolated ablations.
- [x] Run tests and confirm they pass.

### Task 3: Temporal stability scoring and selection

**Files:**
- Create: `experiment_b_suite/stability.py`
- Test: `tests/test_stability.py`

**Interfaces:**
- `binary_auc(values, y) -> float`
- `score_temporal_stability(X, y_binary, semesters, config) -> pandas.DataFrame`
- `select_temporally_stable_features(...) -> tuple[list[str], pandas.DataFrame]`

- [x] Write failing tests for tied-rank AUC, stable-direction retention, direction-reversal rejection, sparse-feature rejection, and protected-domain-feature retention.
- [x] Run tests and confirm correct failures.
- [x] Implement per-semester AUC strength, direction consistency, observed-rate stability, nonconstant-semester rate, and the fixed selection rule.
- [x] Run tests and confirm they pass.

### Task 4: Correlation-cluster pruning

**Files:**
- Create: `experiment_b_suite/redundancy.py`
- Test: `tests/test_redundancy.py`

**Interfaces:**
- `prune_correlated_features(X_train, selected, stability_report, threshold) -> tuple[list[str], dict]`

- [x] Write failing tests showing that highly correlated features form one cluster and that the representative with the higher temporal priority score is retained.
- [x] Run tests and verify failure.
- [x] Implement train-only median-imputed Pearson correlation, union-find clustering, and deterministic representative selection.
- [x] Run tests and confirm they pass.

### Task 5: Missingness-aware, winsorized, and quantile-rank preprocessors

**Files:**
- Create: `experiment_b_suite/preprocessing.py`
- Test: `tests/test_preprocessing.py`

**Interfaces:**
- `fit_transform_preprocessor(train, test, mode, ...) -> PreprocessingResult`
- Supported modes: `baseline`, `missing_aware`, `winsor`, `quantile_rank`.

- [x] Write failing tests showing that baseline reproduces mean-impute/z-score behavior, missing-aware uses observed-only variance, winsor bounds come only from training values, and quantile-rank test mapping uses the training empirical distribution.
- [x] Run tests and verify failure.
- [x] Implement the four frozen preprocessing modes without adding duplicate transformed copies.
- [x] Run tests and confirm they pass.

### Task 6: Sparse-history masks and course-status compression

**Files:**
- Create: `experiment_b_suite/representation.py`
- Test: `tests/test_representation.py`

**Interfaces:**
- `add_sparse_history_masks(train, test) -> FeatureTransformResult`
- `compress_course_status(train, test) -> FeatureTransformResult`

- [x] Write failing tests for same-course, target-course, component, recent-component, and conduct availability masks.
- [x] Write failing tests proving that 0/1/2 course-status columns are replaced by compact counts, proportions, nonzero count, mean, standard deviation, and entropy.
- [x] Run tests and verify failure.
- [x] Implement the representations.
- [x] Run tests and confirm they pass.

### Task 7: Chronological cross-fitted risk bins

**Files:**
- Create: `experiment_b_suite/risk_bins.py`
- Test: `tests/test_risk_bins.py`

**Interfaces:**
- `build_cross_fitted_risk_bins(train_raw, test_raw, candidates, config) -> FeatureTransformResult`

- [x] Write failing tests demonstrating that changing a future-semester training label cannot change an earlier-semester risk feature, same-semester labels are excluded, test features use all training semesters, missing values receive a smoothed missing-bin risk, and the first semester receives `0.05`.
- [x] Run tests and verify failure.
- [x] Implement expanding-semester quantile edges, smoothed bin risks, missing-bin handling, and train/test mapping.
- [x] Run tests and confirm they pass.

### Task 8: Experiment feature orchestration

**Files:**
- Create: `experiment_b_suite/features.py`
- Test: `tests/test_features.py`

**Interfaces:**
- `build_experiment_features(train_raw, test_raw, spec, config) -> FeatureBuildResult`

- [x] Write failing tests for B0 exact preservation, B3 stability-only selection, B4 stability plus correlation pruning, B5 B4 plus masks, B6 branches from B4 rather than B5, B7 replaces course-status columns, and B8 adds only chronological risk-bin features to B4.
- [x] Run tests and verify failure.
- [x] Implement the experiment pipelines and detailed manifests.
- [x] Run tests and confirm they pass.

### Task 9: Runner, CLI, outputs, and analysis

**Files:**
- Create: `experiment_b_suite/runner.py`
- Create: `experiment_b_suite/error_analysis.py`
- Create: `run_experiment_b.py`
- Create: `inspect_predictions.py`
- Create: `run_smoke.sh`
- Create: `run_dev_all.sh`
- Create: `run_holdout_finalists.sh`
- Test: `tests/test_runner.py`

**Interfaces:**
- `run_many_experiments(...) -> pandas.DataFrame`
- Output files include comparison, fold metrics, equal-budget metrics, training diagnostics, predictions, feature manifests, temporal stability reports, and pruning reports.

- [x] Write failing tests for CLI experiment parsing, separate output paths, development-versus-holdout tuning semantics, and manifest persistence.
- [x] Run tests and verify failure.
- [x] Implement the runner and scripts by preserving Track A's policy and metric behavior.
- [x] Run tests and confirm they pass.

### Task 10: Documentation and verification

**Files:**
- Create: `README.md`
- Create: `VERIFICATION.md`
- Create: `requirements.txt`
- Create: `verification/*`

- [x] Document every experiment, leakage boundary, expected output, development command, holdout-finalist command, and interpretation guidance.
- [x] Run `python -m pytest -q` and save the output.
- [x] Run `python -m compileall -q experiment_b_suite run_experiment_b.py inspect_predictions.py` and save the status.
- [x] Run a real Information Technology fold-1 smoke test for all experiments with one epoch.
- [x] Verify B0 has 276 features on the actual fold and compare its smoke metrics with the prior exact baseline.
- [x] Run ZIP integrity verification and save the result.
