# Track A Academic Early-Warning Experiment Suite

This package implements Track A using only the existing expanding-fold CSVs. It does **not** reconstruct features from raw database tables.

The central constraint is preserved throughout:

- `IDSV`, `IDMH`, registration identifiers, target fields, and metadata are excluded from neural-network inputs.
- All 276 original E0 features are retained in every experiment.
- A2–A8 add transformations of the upstream historical features already present in each target row.
- Every prior, imputation value, clipping boundary, category vocabulary, and scaling statistic is fitted from the current fold's training rows only.
- The MultiTaskNet architecture, effective-number classification weighting, regression weighting, optimizer, and loss weights remain unchanged.

## Experiment ladder

| Code | Experiment | Change from the previous stage |
|---|---|---|
| A0 | `baseline_exact` | Exact E0 target filtering, features, preprocessing, and model |
| A1 | `grade10_only_filter` | Requires `target_grade10` only instead of also requiring `target_grade4` |
| A2 | `history_reliability` | Adds history-depth bins and support/reliability coefficients |
| A3 | `support_shrinkage` | Adds train-only empirical-Bayes shrinkage for low-support rates |
| A4 | `recent_long_term_trends` | Adds recent-minus-long-term academic trends |
| A5 | `risk_ratios_thresholds` | Adds failed-credit ratios, GPA thresholds, workload deltas, and warning interactions |
| A6 | `ability_course_compatibility` | Adds ability-versus-course-difficulty gaps using upstream features |
| A7 | `family_missingness` | Adds feature-family history-availability indicators |
| A8 | `semantic_preprocessing` | Adds log-count channels, train-only winsorization, unscaled binary flags, and one-hot course clusters |

The experiments are cumulative so that each transition has a clear interpretation. A0 is always included as the control.

## Important result of the target-filter audit

On the supplied IT folds, A1 currently recovers no additional valid rows:

- folds 1–5: no valid `target_grade10` rows are missing `target_grade4`;
- fold 6 test: one row is missing `target_grade10` itself;
- fold 7 train: one row is missing `target_grade10` itself.

A1 should therefore reproduce A0 on this dataset. It remains in the suite to document and verify the data-construction assumption.

## Installation

```bash
unzip track_a_experiment_suite.zip
cd track_a_experiment_suite
python3 -m pip install -r requirements.txt
```

For CUDA-enabled PyTorch, install the suitable PyTorch build for the server before installing the remaining requirements.

## 1. Run the target-filter audit

```bash
python3 audit_target_filter.py \
  --faculty khoa_cong_nghe_thong_tin \
  --splits_dir /your/path/data/splits/expanding \
  --output target_filter_audit.csv
```

## 2. Smoke-test all A0–A8 paths

One target fold, one epoch:

```bash
python3 run_track_a.py \
  --faculty khoa_cong_nghe_thong_tin \
  --splits_dir /your/path/data/splits/expanding \
  --output_dir ./track_a_smoke_outputs \
  --phase dev \
  --dev_folds 1 \
  --experiments all \
  --epochs 1 \
  --batch_size 512 \
  > track_a_smoke.log 2>&1
```

Equivalent shell helper:

```bash
SPLITS_DIR=/your/path/data/splits/expanding ./run_smoke.sh
```

## 3. Run the complete development study

```bash
python3 run_track_a.py \
  --faculty khoa_cong_nghe_thong_tin \
  --splits_dir /your/path/data/splits/expanding \
  --output_dir ./track_a_outputs_dev \
  --phase dev \
  --dev_folds 1,2,3,4,5 \
  --experiments all \
  --epochs 100 \
  --batch_size 512 \
  > track_a_dev.log 2>&1
```

Equivalent helper:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
OUTPUT_DIR=./track_a_outputs_dev \
./run_dev_all.sh
```

This executes 9 experiments × 5 folds = 45 model-training runs.

## 4. Select development finalists

Primary evidence:

- pooled classification AP;
- mean and worst-fold AP;
- Recall@10%, Recall@15%, and Recall@20%;
- F2 at equal warning budgets;
- fixed-threshold and drift-aware confusion matrices.

Do not send all A0–A8 automatically to holdout. Keep A0 and select only a small number of defensible development finalists.

## 5. Run holdout finalists

```bash
python3 run_track_a.py \
  --faculty khoa_cong_nghe_thong_tin \
  --splits_dir /your/path/data/splits/expanding \
  --output_dir ./track_a_outputs_holdout \
  --phase holdout \
  --dev_folds 1,2,3,4,5 \
  --holdout_folds 6,7 \
  --experiments A0,A3,A6 \
  --epochs 100 \
  --batch_size 512 \
  > track_a_holdout.log 2>&1
```

The example finalists above are placeholders for the experiments that actually win development selection; they are not preselected recommendations.

Equivalent helper after setting the finalists:

```bash
SPLITS_DIR=/your/path/data/splits/expanding \
EXPERIMENTS=A0,A3,A6 \
OUTPUT_DIR=./track_a_outputs_holdout \
./run_holdout_finalists.sh
```

## 6. Inspect prediction errors

```bash
python3 inspect_predictions.py \
  ./track_a_outputs_holdout/khoa_cong_nghe_thong_tin/A6_ability_course_compatibility \
  --warning_col warn_at_15pct \
  --output A6_error_analysis_15pct.csv \
  > A6_error_analysis_15pct.log 2>&1
```

Supported warning columns in saved prediction files include:

- `warn_fixed`
- `warn_drift`
- `warn_at_5pct`
- `warn_at_10pct`
- `warn_at_15pct`
- `warn_at_20pct`

The report groups errors by fold, grade severity, actual historical-semester depth, and target-course history availability.

## Output layout

```text
track_a_outputs_dev/
└── khoa_cong_nghe_thong_tin/
    ├── experiment_comparison_dev.csv
    ├── A0_baseline_exact/
    │   ├── predictions_fold_1.csv
    │   ├── ...
    │   ├── fold_metrics_dev.csv
    │   ├── equal_budget_metrics_dev.csv
    │   ├── target_filter_audit_dev.csv
    │   ├── training_diagnostics_dev.csv
    │   ├── feature_manifest_dev.json
    │   └── summary_dev.json
    ├── A1_grade10_only_filter/
    └── ...
```

## A3 shrinkage strength

The default is:

```text
alpha = 10
```

A separate controlled sensitivity study can use:

```bash
--shrinkage_alpha 5
--shrinkage_alpha 10
--shrinkage_alpha 25
```

Do not mix several alpha values into the first A0–A8 comparison. Run the default ladder first, then tune alpha only when A3 or a later dependent experiment shows development value.

## Verification commands

```bash
pytest -q
python3 -m py_compile run_track_a.py audit_target_filter.py inspect_predictions.py track_a_suite/*.py
```
