# Current best (inventory 2026-10-26)

Do not train from this file. It only records what already exists.

## Hardware

`nvidia-smi` reports **1 GPU**: Tesla V100-SXM2-32GB, 32768 MiB (index 0).
Only one heavy training job at a time. Take the lock before any GPU training:

```bash
exec 9>/root/disk_3tb/eeg-thao/multi_agent_opt/gpu.lock
flock -n 9 || { echo "GPU busy"; exit 1; }
```

## Metric definitions (project's own)

Two different numbers are both called accuracy. The targets use the official ones.

| Symbol | Definition | How it is stored |
|---|---|---|
| **Acc** | Subject-mean sample accuracy of the 128 Hz decision after the HMM. Each subject is scored, then subjects are averaged. Not balanced accuracy. | CSV column `Acc`, **percent** (82.42 means 82.42%) |
| **L** | For each attention switch, find the first time the prediction flips to the new label and holds it for at least 0.5 s inside a 100 s window. Switch score = exp(−SDT/60). Undetected = 0. Per subject: median over switches. Then mean over subjects. | fraction in \[0, 1\] |
| **H** | Harmonic mean `2 * (Acc/100) * L / ((Acc/100) + L)` | CSV column `H`, **fraction** (0.8677 means 86.77). Target "h >= 85" means **H >= 0.85** |
| acc1 | Median-over-subjects accuracy of non-overlapping 1 s windows of the raw evidence, before the official aggregator | **not** the target |
| SDT | Median detection delay in seconds | seconds |

Code:

- Official port: `/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/reports/eda_full/scripts/scorer.py` (`fast_score_subject`, `aggregate`).
- LODO wrapper used by every transformer result: `/root/disk_3tb/eeg-thao/score.py` (`score_dataset`, `score_experiment`). Synthetic switches, identical schedule to the EDA (`e1_switch.synth_trial`). HMM emissions fit on the other datasets only. Lookahead for the logit models is **0.75 s**.
- Real-switch external scoring (AASD key-press, Carta cues) is inside `eval_external.score`, called by `ensemble_clf_transformer_v4.py` and by `external_train_v2.py`. Same H and Acc. Labels of AASD and Carta are not used to fit weights or to pick the headline HMM pair `(p_stay=0.994, scale=0.8)`.

Python: `/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/.venv/bin/python`

## The 5 training datasets

Leave-one-dataset-out. Keys in `common.DS_KEYS`, labels in `common_exp.LAB`:

| Key | Corpus |
|---|---|
| AVGC | KUL-AV-GC |
| DTU | DTU |
| KUL | KUL2016 |
| UHD | KUL-UHD |
| USTC | USTC |

**Mean vs per dataset.** The project always stores both. The number the transformer logs call the goal (`experiments/transformer_v2.log`: `goal H>0.85 and Acc>80`) is the **unweighted mean of the five held-out folds**. The user also asked for the five datasets themselves. Both are below. The mean target is already met. The per-dataset target is not: AVGC misses H and Acc, UHD misses H by 0.009.

## Best train system

**Honest LODO mean best (2026-10-27 addendum):** IRM↔SWA logit blend at fixed **w=0.5** (frozen `stem_ms96_irm` + `stem_ms96_swa`; no retrain). Mean H **0.8725**, Acc **82.58**. AVGC **0.8100 / 75.14** (still open vs ≥0.85/78). UHD H **0.8580 MET**. AASD Acc ~50.9 (below `linear_hp30`). See `multi_agent_opt/results_irm_swa_fuse.md`.

**Nested-LODO architecture baseline:** **`stem_ms96`**, seed `20270101`, `STEPS=1000`, batch 512.
One architecture on every fold (not the acc74 mixture). HMM chosen on inner datasets only, then the network is refit on the four non-held-out datasets and the held-out dataset is scored once.

Checkpoints: `/root/disk_3tb/eeg-thao/models/logit_tf_v2/stem_ms96__held-{AVGC,DTU,KUL,UHD,USTC}.pt`
Table: `/root/disk_3tb/eeg-thao/results/research/ensemble_clf_transformer_v2.csv` rows `role=family_stem_ms96`
Log: `/root/disk_3tb/eeg-thao/experiments/transformer_v2.log`

| Held-out | H | Acc % | p_stay | scale | vs H>=0.85 | vs Acc>=78 |
|---|---:|---:|---:|---:|---|---|
| AVGC | 0.7945 | 74.06 | 0.994 | 0.8 | miss 0.0555 | miss 3.94 |
| DTU | 0.9061 | 89.85 | 0.994 | 0.8 | met | met |
| KUL | 0.8755 | 82.33 | 0.994 | 0.8 | met | met |
| UHD | 0.8407 | 79.47 | 0.994 | 0.8 | miss 0.0093 | met |
| USTC | 0.9219 | 86.36 | 0.996 | 0.8 | met | met |
| **mean** | **0.8677** | **82.42** | | | **met** | **met** |

Pre-registered selector `acc74` (pick model and HMM from inner scores only; DTU selected `ens_all`, the other four selected `stem_ms96`) is the honest single submission rule. It is worse on the mean: **H 0.8562, Acc 82.75**. Same file, `role=acc74`. It also meets the mean bars and fails the same per-dataset cells (DTU Acc is 91.53 but H falls to 0.8485).

`ens_gate_local_stem` (`transformer_v3_screen.csv`) is a different, cheaper protocol (validation subjects, not a full inner dataset). Mean H 0.8676, Acc 82.75. It does not beat `stem_ms96` on mean H. Do not treat it as a new best.

## Best external (AASD and Carta2026)

Real switches. Headline HMM is the pre-specified pair from the train acc74 rule, **not** a sweep on the test labels.

**Best external readout: `linear_hp30`.** Causal one-pole high-pass (tau 30 s) of the four direct-classifier seed logits, then their mean. No new weights.
Table: `/root/disk_3tb/eeg-thao/results/research/ensemble_clf_transformer_v4_external.csv`
Log: `/root/disk_3tb/eeg-thao/experiments/transformer_v4_external.log`

| Set | H | Acc % | acc1 | p_stay | scale |
|---|---:|---:|---:|---:|---:|
| AASD | 0.6422 | 53.12 | 0.536 | 0.994 | 0.8 |
| Carta2026 | 0.6520 | 60.54 | 0.613 | 0.994 | 0.8 |

Same high-pass on the five train folds falls to **mean H 0.7899, Acc 69.21** (`ensemble_clf_transformer_v4_hp_lodo.csv`). It cannot be the train model.

The train winner `stem_ms96` on the same external protocol (preset row, file only in the unzipped tree):

`/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/eeg-thao_unzipped/eeg-thao/results/research/ensemble_clf_transformer_v2_external.csv`

| Set | H | Acc % | acc1 |
|---|---:|---:|---:|
| AASD | 0.6158 | 51.54 | 0.523 |
| Carta2026 | 0.5695 | 51.03 | 0.530 |

### Two readings of the external target

Numbers above are the ones to check. Neither reading is fully met, and the failure is AASD accuracy. Frame accuracy on AASD is about 0.52–0.54, so another HMM or another logit readout cannot reach the 60s.

**Reading 1 (this inventory brief): H > 0.63 and Acc > 60 on each of AASD and Carta.**

| Set | H > 0.63 | Acc > 60 | with `linear_hp30` preset |
|---|---|---|---|
| AASD | yes (0.6422) | **no** (53.12, gap 6.88) | |
| Carta2026 | yes (0.6520) | yes (60.54) | |

**Reading 2 (original wording "assd và carta lần lượt lớn hơn 63, 60", used by the 2026-10-24 run): AASD must clear 63 on both H and Acc; Carta must clear 60 on both H and Acc.**

| Set | bar | H | Acc | result |
|---|---|---:|---:|---|
| AASD | > 63 | 0.6422 (64.22) passes H | 53.12 fails Acc | fail |
| Carta2026 | > 60 | 0.6520 passes | 60.54 passes | pass |

A test-label sweep of `p_stay` on the older 4-decoder + 4-classifier fusion reaches AASD H 0.6525 / Acc 54.03 at `p_stay=0.95` (`fuse_external.csv`, name `RB`). That is the best AASD H on disk and still misses Acc 60 and Acc 63. Do not retune `p_stay` on AASD or Carta. The safe pre-specified operating point remains 0.998 or the transformer preset 0.994/0.8. `p_stay=0.9997` is the worst on real switches.

## Gap

| Target | Status |
|---|---|
| Mean of 5 train folds, H >= 0.85 and Acc >= 78 | **met** by IRM↔SWA w=0.5 (0.8725 / 82.58), by `stem_ms96` (0.8677 / 82.42), and by acc74 (0.8562 / 82.75) |
| Each of the 5 train folds, H >= 0.85 and Acc >= 78 | **not met**. Best AVGC so far: fuse **0.8100 / 75.14** (still short). UHD H **met** at 0.8580 via fuse (stem alone was 0.8407) |
| AASD and Carta, reading 1 (H>0.63 and Acc>60 each) | **not met**. Carta passes on `linear_hp30`. AASD Acc 53.12 |
| AASD > 63 and Carta > 60 on both metrics (reading 2) | **not met**. Carta passes. AASD Acc 53.12 |

**OPEN:** AVGC ≥0.85/78; AASD Acc>60.

No checkpoint meets the train mean and both external sets at once. Fuse wins the train mean and loses AASD Acc (~51). `linear_hp30` wins Carta / AASD Acc among externals and loses the train mean (H 0.79, Acc 69.2).

## Commands that reproduce the recorded results

From `/root/disk_3tb/eeg-thao` unless noted. These rerun finished work. Do not start them to "confirm" unless the claim board says so.

Train and score the five datasets (writes `results/research/ensemble_clf_transformer_v2.csv`, checkpoints under `models/logit_tf_v2/`). The log that matches the table above used **1000** steps, not the script default of 800:

```bash
cd /root/disk_3tb/eeg-thao
export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 OMP_NUM_THREADS=8 MKL_NUM_THREADS=8
STEPS=1000 BATCH=512 \
  /root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/.venv/bin/python -u ensemble_clf_transformer_v2.py
```

Score AASD and Carta for those four v2 recipes. This CLI exists only in the unzipped tree. The workspace copy of `ensemble_clf_transformer_v2.py` has no `--mode`.

```bash
cd /root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/eeg-thao_unzipped/eeg-thao
export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/.venv/bin/python -u \
  ensemble_clf_transformer_v2.py --mode external --stage test
```

External winner (`linear_hp30`, `linear_mean`, `rel_ms`, `rel_hp`) and the train-set high-pass rescore:

```bash
cd /root/disk_3tb/eeg-thao
export PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 OMP_NUM_THREADS=4 MKL_NUM_THREADS=4
MODE=external STEPS=800 \
  /root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/.venv/bin/python -u ensemble_clf_transformer_v4.py
MODE=lodo_hp \
  /root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/.venv/bin/python -u ensemble_clf_transformer_v4.py
```

Older fusion that `README_BEST.md` calls the best legal decoder+classifier stack (mean LODO H about 0.798, not the transformer). Run from `experiments/decoder_lodo` of the dataset repo; full command block is section 12 of `/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/experiments/decoder_lodo/README_BEST.md`.

## What is not the best

- Image-split retrain of the same `stem_ms96` (`stem_ms96_image_split.py`, `STEPS=1000`): mean H 0.8627, Acc 82.50. Below nested `stem_ms96`.
- v1 `tf_hp_w48` acc74: mean H 0.8208, Acc 78.09.
- Nested LDA on the four logits (`ensemble_clf_nested.py`): mean H 0.8065, Acc 72.64.
- 4-decoder + 4-classifier `sw2` fusion: mean LODO H 0.798, Acc about 73. See `README_BEST.md`.

## Live portfolio tip (multi-agent)

Updated: 2026-10-30 (evidential-tip-stack hurt; AVGC tip still causal-ds-temp held_T_nll_med; Acc tip 54.05).

| Role | Method | Key scores |
|------|--------|------------|
| LODO / AVGC tip | **focal↔synth + held_T_nll_med** | mean **0.8844 / 83.38**; AVGC **0.8245 / 75.64**; UHD **0.8630** |
| Prior fuse raw | focal-IRM ↔ synth fuse w=0.5 | 0.8821 / 83.36; AVGC 0.8226 / 75.54 |
| Prior LODO tip | IRM↔synth fuse w=0.5 | 0.8790 / 83.11; AVGC 0.8192 / 75.52 |
| External Acc | tip_fuse_a0.50@SW2emis (speech-xattn↔hard_abs_swmix) | AASD **54.05**; Carta **60.60** |
| External Acc (prior) | SW2↔swmix hard_abs_swmix|t1 | AASD 53.53; Carta 59.71 |
| External Carta | linear_hp30 | AASD 53.12; Carta **60.54** |

Still open: AVGC ≥0.85/78 (gap ≈−0.0255 H); AASD Acc>60.

**CLOSED families:** Mixup/BCE/focal stem re-tunes; switch-schedule FT; EEG noise/SpecAug clf FT; Path A classical Acc; DANN/GRL λ; VICReg/CPC SSL; SpecViT Mel Acc; tip Acc PP; emission-shift Acc PP; speech-xattn-dec; **adapt-readout / causal-ds-temp / multi-τ** decision-layer T on tip fuse; **isotonic-ds-fuse**; **evidential-tip-stack / decision-layer stacking** on tip stems. **No tip-fuse PP: no T/iso/stack/attn/agree.**

**NEXT Acc:** New Acc front-end distinct from closed speech-xattn / temp-attn / speech-resid / aux-env / SpecViT / Path A (claim separately).

**NEXT AVGC:** **group-dro-stem-avgc** — Group-DRO / CVaR worst-dataset stem training from tip inits (train-time logits) — ≠ closed IRM fixed-w / AVGC λ curricula / Mixup / decision-PP. Gap ≈−0.0255 H.
