# Experiment B — Temporally Robust Feature Engineering

This package evaluates whether the accepted E0 neural network can learn more reliably from a cleaner, smaller, and temporally more stable representation of the **existing fold CSV features**.

It does **not** change:

- `MultiTaskNet` architecture;
- classification, regression, or blended heads;
- effective-number weighting;
- regression/blend low-grade weighting;
- AdamW settings;
- random seed;
- expanding-fold protocol;
- E0 target-row filtering;
- exclusion of `IDSV`, `IDMH`, targets, identifiers, and target-semester metadata from network inputs.

It does **not** reconstruct features from `data1.zip` or `data2.zip`.

## Scientific motivation

Track A mainly added transformations to the already engineered 276-column E0 representation. Its development improvements did not generalize to folds 6–7. Experiment B tests a different hypothesis:

> The baseline may be limited less by missing derived features than by noisy, redundant, sparse, and temporally unstable representations.

Every selector and preprocessor is fit independently inside each fold using only that fold's training rows. Fold-test labels never influence feature construction.

## Experiments

| Code | Representation | Relationship to E0 |
|---|---|---|
| `B0` | Exact E0 baseline | Control; 276 inputs on the verified IT fold 1 |
| `B1` | Structural cleaning | Removes train-all-null, train-constant, exact duplicates, unstable registration timing, and absolute semester fields |
| `B2_CS` | Course-status ablation | Removes only `course_status_*` |
| `B2_TIME` | Time-field ablation | Removes only unstable registration timing and absolute semester fields |
| `B2_CLUSTER` | Cluster-ID ablation | Removes only numeric `course_cluster_id` |
| `B2_REDUNDANT` | Isolated redundancy pruning | Starts from E0 and prunes correlations `>=0.995`, choosing representatives by train-only temporal utility |
| `B3` | Temporal stability selection | Keeps features with sufficient support, temporal strength, and predictive-direction consistency; preserves a protected domain-core list |
| `B4` | Stable nonredundant core | `B3` plus correlation-cluster pruning at `|r|>=0.95` |
| `B5` | Sparse-history representation | `B4` plus targeted history-availability masks and observed-only standardization |
| `B6_W` | Winsorized core | `B4` with train-only 0.5%/99.5% winsorization replacing the raw continuous representation |
| `B6_Q` | Quantile-rank core | `B4` with train-empirical percentile ranks replacing continuous values |
| `B7` | Course-status compression | `B4` plus compact neutral summaries of the 0/1/2 status categories; original 140 columns are removed |
| `B8` | Cross-fitted risk bins | `B4` plus chronological, smoothed risk-bin features for seven strong variables |

The later experiments branch from `B4`; they are not one cumulative ladder:

```text
B3 -> B4 -> B5
         -> B6_W
         -> B6_Q
         -> B7
         -> B8
```

## Temporal stability selection

For each feature and each training semester, the suite calculates a tie-aware univariate AUC. It then measures:

- observed-row rate;
- fraction of training semesters in which the feature is nonconstant;
- median `|AUC - 0.5|`;
- number of valid semesters;
- predictive-direction consistency;
- missing-rate instability;
- a deterministic priority score used for correlation-cluster representatives.

Default selection criteria:

```text
observed rate                         >= 0.10
nonconstant training-semester rate   >= 0.80
valid training semesters             >= 2
median |AUC - 0.5|                   >= 0.02
direction consistency                >= 0.70
```

A small domain-critical list is retained whenever present, including GPA, history depth, failed credits, warning history, component grades, conduct, target-course history, same-course history, and workload.

Selection is fold-specific. A feature retained in fold 1 may be rejected in fold 5 if its temporal behavior changes.

## B8 leakage boundary

For a training row in semester `s`, each B8 risk-bin feature uses:

```text
feature values and failure labels from training semesters < s only
```

It does not use labels from the row's own semester. The first training semester receives the fixed prior `0.05`. For a fold-test row, the encoding uses all eligible rows in that fold's training set.

Default risk-bin settings:

```text
5 train-fitted quantile bins
empirical-Bayes alpha = 20
initial prior = 0.05
minimum prior-history rows = 50
```

Candidate variables:

```text
prev_semester_gpa10
prev_cumulative_gpa10
hist_failed_credits_sum
hist_warning_rate
course_fail_rate_clipped
hist_target_course_mean_grade10
hist_same_course_last_grade10
```

## Installation

```bash
unzip experiment_B.zip
cd experiment_B
python3 -m pip install -r requirements.txt
```

Run commands from the package root so Python can import `experiment_b_suite`.

## 1. One-fold smoke test

The helper executes each experiment in a separate Python process. This limits memory accumulation and makes the run resumable.

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_smoke_outputs \
EPOCHS=1 \
./run_smoke.sh
```

By default it uses development fold 1 and all experiments.

To smoke-test only selected variants:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_smoke_outputs \
EXPERIMENTS=B0,B3,B4,B5,B6_Q,B8 \
EPOCHS=1 \
./run_smoke.sh
```

Completed experiments are skipped when their summary exists. Force reruns with:

```bash
SKIP_EXISTING=0 ... ./run_smoke.sh
```

## 2. Full development study on folds 1–5

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_outputs_dev \
EPOCHS=100 \
BATCH_SIZE=512 \
./run_dev_all.sh
```

This is:

```text
13 experiments × 5 folds = 65 model-training runs
```

A narrower first pass can be executed with:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_outputs_dev_core \
EXPERIMENTS=B0,B1,B2_CS,B2_TIME,B2_CLUSTER,B2_REDUNDANT,B3,B4 \
EPOCHS=100 \
./run_dev_all.sh
```

Then execute the B4 branches:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_outputs_dev_branches \
EXPERIMENTS=B0,B4,B5,B6_W,B6_Q,B7,B8 \
EPOCHS=100 \
./run_dev_all.sh
```

Using one output root for all experiments is more convenient for final aggregation. The two-pass examples are primarily useful when compute must be staged.

## Direct Python command

A single experiment:

```bash
python3 run_experiment_b.py \
  --faculty khoa_cong_nghe_thong_tin \
  --splits_dir /your/path/data/splits/expanding \
  --output_dir ./experiment_B_outputs_dev \
  --phase dev \
  --dev_folds 1,2,3,4,5 \
  --experiments B4 \
  --epochs 100 \
  --batch_size 512 \
  > B4_dev.log 2>&1
```

Multiple experiments are accepted by the CLI, but the shell helpers deliberately isolate experiments into separate processes.

## 3. Development selection

Primary decision criteria:

1. `classification_ap`;
2. `classification_mean_fold_ap`;
3. `classification_worst_fold_ap`;
4. `classification_recall_at_15pct`;
5. `classification_f2_at_15pct`;
6. fold-by-fold consistency;
7. feature count as a complexity tie-breaker.

Do not promote a candidate solely because its development-selected threshold produces more warnings.

The combined development table is:

```text
experiment_B_outputs_dev/
└── khoa_cong_nghe_thong_tin/
    └── experiment_comparison_dev.csv
```

## 4. Diagnostic holdout procedure

Folds 6–7 have already been inspected repeatedly. They should be described as reused temporal diagnostics rather than pristine final holdouts.

After selecting two or three development finalists, always include `B0`:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./experiment_B_outputs_holdout \
EXPERIMENTS=B0,B4,B8 \
EPOCHS=100 \
BATCH_SIZE=512 \
./run_holdout_finalists.sh
```

`B0,B4,B5,B6_Q` in the helper is an example default, not a preselected scientific conclusion. Override `EXPERIMENTS` using the development evidence.

## 5. Error analysis

Example equal-budget analysis:

```bash
python3 inspect_predictions.py \
  ./experiment_B_outputs_holdout/khoa_cong_nghe_thong_tin/B8_cross_fitted_risk_bins \
  --warning_col warn_at_15pct \
  --output B8_error_analysis_15pct.csv \
  > B8_error_analysis_15pct.log 2>&1
```

The output groups TP/FN/FP/TN by:

- fold;
- grade band;
- actual historical-semester depth (`0`, `1`, `2`, `3–5`, `6+`);
- target-course history availability.

It does not use supervised `IDSV` overlap as a proxy for real historical availability.

## Output structure

Each experiment directory contains:

```text
summary_<phase>.json
fold_metrics_<phase>.csv
equal_budget_metrics_<phase>.csv
training_diagnostics_<phase>.csv
target_filter_audit_<phase>.csv
feature_manifest_<phase>.json
predictions_fold_<n>.csv
```

Experiments involving stability selection also contain:

```text
temporal_stability_fold_<n>.csv
```

Experiments involving correlation pruning also contain:

```text
correlation_pruning_fold_<n>.json
```

The feature manifest records:

- exact retained features;
- features removed from E0;
- features added to E0;
- fold-specific preprocessing parameters;
- winsor bounds or quantile support;
- B8 bin edges, priors, and smoothed risks;
- paths of saved selection/pruning artifacts.

## Custom selection parameters

```bash
python3 run_experiment_b.py \
  ... \
  --experiments B3,B4 \
  --stability_min_strength 0.02 \
  --stability_direction_consistency 0.70 \
  --correlation_threshold 0.95
```

B8 parameters:

```bash
python3 run_experiment_b.py \
  ... \
  --experiments B8 \
  --risk_bin_count 5 \
  --risk_bin_alpha 20
```

Change these only through development folds. Do not tune them using folds 6–7.
