# Context: Track 03 Sleep Onset (NeurIPS 2026 EEG/EMG Foundation Challenge)

Snapshot written 2026-10-15, updated 2026-10-16 (E1 filters, distributional head, E8, seed-42 public test). Purpose: let a person or a new assistant session pick the work up without re-deriving anything. Everything marked "measured" was computed in this session; everything marked "not verified" was not.

Project root: `/root/disk_3tb/Neural_Interfaces_2026_Track_3_Sleep_Onset`

## 1. Task

- **Goal:** from 4-channel Muse EEG (TP9, AF7, AF8, TP10; 128 Hz; microvolts, unfiltered), predict for every non-overlapping 5 s window the seconds remaining until the first N2 epoch, capped at 600 s.
- **Inference contract:** windows arrive one at a time in time order (B = 1), from the start of the recording until first N2. A model may keep state within a recording (`reset_state()` is called between recordings) but may never use future windows, the recording length, the order of recordings, or participant identity. Elapsed time inferred from the number of windows received is allowed.
- **Metric, W-bMAE (seconds, lower is better):** inside each recording, MAE within the target bins [0,40), [40,90), [90,300), [300,600] with weights 10, 5, 3, 1, averaged over the non-empty bins; then averaged over recordings.
  - Sealed phase: computed separately for **seen** (new nights of participants present in training) and **unseen** participants, then macro-averaged 50/50.
  - Warm-up phase (public): 40 public test recordings, all from participants also in train; indicative only.
- **Official code** lives in the user's clone `2026-competition/` (benchopt benchmark, `tracks/sleep_onset/`). Do not add generated files there.

## 2. User preferences

- Chat replies in **Vietnamese** (with diacritics); code, file names and technical terms stay English.
- Training is expected to use the **GPU**; flag any CPU-only run.
- Keep generated files out of `2026-competition/`.
- **Public test split: always seed 42** (the organizers' `seed: 42` in `training.yml`). Never pick a model or seed by its public-test score. CV may use seeds 0, 1, 2 to measure variance. The user also dropped the idea of writing a benchopt solver for the seed runs ("just run the same seed as the organizers").
- The user asks short, conceptual questions ("what is ICC", "what is hazard"); answer plainly with a concrete example from this dataset.

## 3. Environment

| Item | State |
|---|---|
| Python for all experiments | `/root/miniconda3/bin/python` 3.13: torch 2.8 (cu126), xgboost 3.1.2, scikit-learn 1.9.1, benchopt, neuralbench 0.4.0, braindecode 1.8.1, mne, reportlab, pypdfium2 |
| Project `.venv` | exists but is **empty** (no numpy, no torch) |
| GPU | Tesla V100 32 GB (shared host, ~23 GB RAM, OOM kills have happened). cu128/cu130 torch wheels have no V100 kernels |
| Public data | `2026-competition/tracks/sleep_onset/data/neural_compet/Interaxon2026Muse/download/nm000287` (BrainVision) |
| **Codabench image** | `lionelyneuro/neural-compet:v3`, built only from `2026-competition/requirements.txt`; nothing is installed at evaluation; only `submission.py` plus weight files are allowed (no extra modules) |
| **xgboost / lightgbm / catboost** | **not in the image** (checked the full dependency closure). A submission may use torch, scikit-learn, numpy, scipy, pandas, mne, braindecode |

## 4. What the data looks like (measured, `eda_muse/`)

- 540 recordings, 203 participants, 157.52 h. Public split: 500 train, 40 test; each of the 40 test participants has exactly one other night in train.
- **Label leak:** `recording_duration − n2_onset = 300 s` for all 540 recordings, so file length gives the answer. Never use file length, sample count, or whole-recording statistics.
- Onset N2: median 690 s, mean 750 ± 350 s, range 60 to 1500 s; 39.4% of recordings have onset < 600 s. ICC(1) of onset = 0.38 (SD between participants ~216 s, between nights of the same person ~275 s).
- Windows: 80,975 evaluation windows (up to first N2). 30.2% have target exactly 600 (cap). Bin share of windows 5.3 / 6.7 / 27.3 / 60.7%, but share of metric weight 52.9 / 26.5 / 15.8 / 4.8%. One window in 0-40 s weighs about 100 windows in 300-600 s.
- **Constant baseline:** always predicting the weighted median, **35 s**, gives W-bMAE 60.7 on the public test (CV 62.1). 0 s gives 77.6; 100 s gives 84.9; 600 s gives 522.4.
- Signal quality: temporal channels have 2x the amplitude of frontal ones; 519/540 recordings touch the ~780 µV ADC rail; 67.8% start with a ~0 sample (switch-on transient); fraction of windows with a bad channel falls from 31% (more than 10 min before N2) to 3.5% (last 40 s), so artifacts are a wakefulness signal, do not clean them away.
- Predictive structure: "late movers" (sigma/beta, 1/f slope, theta/alpha) change in the last 1.5-2.5 min; "slow drifters" (beta, gamma, alpha, amplitude, artifacts) drift over 5-10 min. Within the last 90 s no step exists at 40 s, single-window AUC for 0-40 vs 40-90 s is at most ~0.67.
- Elapsed time alone has Spearman -0.44 with the target, but this is largely a property of how the public set was curated (all N2 within ~25-30 min; hazard per minute rises to 100% at minute 24). Whether the sealed set is cut the same way is unknown.
- 77 "off-grid" recordings (onset not a multiple of 30 s, onset >= 914 s, no start-up transient): 13.6% of train, 22.5% of test. First minute of test recordings is cleaner than train (p = 0.0014).

## 5. Files

| Path | Content |
|---|---|
| `eda_muse/EDA_Muse_va_phuong_phap.md` / `.pdf` | Vietnamese report (16 pages): sections 1-10 EDA, 11 modelling implications, 12 foundation models, 13 other models and heads, 14 datasets, 15 papers, 16 preprocessing, 17 augmentation, 18 feature engineering. All links verified by web search on 2026-10-14 unless noted |
| `eda_muse/scripts/` | `01_extract_features.py`, `02_analyze.py`, `03_md_to_pdf.py` (Markdown to PDF) |
| `PREPROCESS_EXPERIMENTS.md` | planned preprocessing experiments E0-E11 with protocol and an empty results table. **None has been run** |
| `experiments/stft_tree/` | STFT features + XGBoost; `README.md` explains everything in detail, includes a glossary |
| `experiments/cnn_gru_feat/` | CNN + GRU on raw EEG and the 217 STFT features |
| `experiments/cvharness.py` | shared 5-fold seen/unseen harness used by the three experiments below |
| `experiments/iir_filter/` | E1: causal IIR low-pass 45 Hz and high-pass 0.1/0.3/0.5 Hz on `cnn_feat_gru` |
| `experiments/dist_head/` | distributional head (121 classes, median decoding), variants `dist_w` and `dist_u` |
| `experiments/e8_bipolar/` | E8: bipolar channels and the 0.1-1 Hz band, variants `e8a`, `e8b`, `e8c` |
| `experiments/public_seed42/` | one run per variant on the public split with seed 42 (headline public numbers) |
| `CNN_GRU_solver.md` | specification and code of the competition CNN+GRU solver (earlier session) |
| `label_report/N2_label_trial_issues.pdf` | report on N2 label and trial-length issues (earlier session) |
| `EDA và SOTA sleep onset.pdf` | Vietnamese deep-research report restored by the user |
| `EXPERIMENTS_LOG.md` | **belongs to a different task** (auditory attention decoding); not relevant here |
| `2026-competition/` | the user's repo clone; `tracks/sleep_onset/solvers/cnn_gru.py` is the existing CNN+GRU solver |

The English translation of the Vietnamese EDA report was started and never finished; its `report_en/` folder no longer exists.

## 6. Experiments and results (measured)

### 6.1 Evaluation protocol shared by both experiments

- 5 participant-grouped folds, seed 0. In each fold: **unseen** = every night of ~1/5 of participants; **seen** = one held-out night of multi-night participants whose other nights stay in train. Fold score = mean(seen, unseen) W-bMAE, mimicking the sealed macro-average.
- 15% of training participants are held out as an inner validation set; the official `binned_mae` from `benchmark_utils/metrics.py` does the scoring.
- 3 seeds. Paired per-recording differences between variants (same fold, seed, recording).
- "Crop-start" probe: drop a random 0-50% of the first windows of each held-out recording and restart context; measures reliance on the clock. Only comparable between variants in the same column.
- Public split: train on the 500 train recordings, score the 40 test recordings.

### 6.2 STFT features + XGBoost (`experiments/stft_tree`, 9.6 min on GPU)

- 217 per-window features: STFT (n_fft 256 = 2 s, hop 64, Hann, 7 frames per 5 s window): log band power and relative power for 6 bands, theta/alpha, sigma/beta, delta/beta, 1/f slope, sigma residual, spectral entropy, SEF50/95, alpha peak, frame-to-frame band variability, amplitude, line length, clip fraction, bad-channel count, 4 channel correlations, 1 Hz spectrogram bins for the TP pair and the AF pair.
- 210 causal context features (EMA 30/120/300 s, EMA30 minus running mean, EMA30 minus EMA300 of 42 core features). A check confirms features of a truncated recording equal the prefix of the full one.
- XGBoost `reg:absoluteerror`, per-window weights equal to the metric, base score = weighted median, early stopping on the inner validation people, then refit.

| Variant | Seen | Unseen | Macro (CV) | Public 40 |
|---|---|---|---|---|
| const (35 s) | 62.2 | 62.0 | 62.1 | 60.7 |
| freq (window only) | 50.7 | 50.7 | 50.7 | 51.7 |
| freq + context | 46.1 | 47.3 | 46.7 | 50.9 |
| + elapsed time | 45.9 | 47.2 | 46.5 | 51.2 |

Context helps (-3.7 ± 0.3 s paired); elapsed time adds nothing (-0.1 ± 0.1). MAE in the 0-40 s bin is 21.8 s, worse than the constant (17.2 s).

### 6.3 scikit-learn check (CPU)

`HistGradientBoostingRegressor(loss="absolute_error")` on the same features, public split, 3 seeds: 50.4 / 51.7 / 49.7 (XGBoost freq+context: 51.0 / 51.1 / 50.6). So a tree model is submittable without xgboost; ~8-11 min per fit on CPU.

### 6.4 CNN + GRU with raw EEG and 217 features (`experiments/cnn_gru_feat`, 29.5 min on GPU)

Encoder: raw window through the solver's 4-block 1-D CNN (GroupNorm, mean + max pooling) concatenated with an MLP over the 217 features standardised with constants fitted on the fold's training recordings; fused to 96-d; unidirectional GRU (128); head `600 * sigmoid(.)`; loss = per-recording W-bMAE; AdamW, OneCycle, 25 epochs, best epoch on the inner validation set (no refit). A start-up check confirms whole-sequence output equals B = 1 streaming (max diff 6e-5 s).

| Model | Seen | Unseen | **Macro (CV)** | **Public 40** (3 seeds) |
|---|---|---|---|---|
| `cnn_gru` (raw EEG, existing solver, 318,945 params) | 48.5 | 48.6 | 48.5 ± 3.0 | 52.6 |
| `feat_gru` (217 features only) | 41.7 | 43.5 | 42.6 ± 1.4 | 49.5 |
| **`cnn_feat_gru` (raw + features)** | **41.7** | **42.6** | **42.2 ± 2.2** | **47.9** (47.7 / 46.4 / 49.7) |

MAE per bin for `cnn_feat_gru`: 15.4 / 35.5 / 83.2 / 294.0 s (0-40 s is below the constant's 17.2 s for the first time). Paired differences: -6.2 ± 0.4 s vs `cnn_gru`, -4.7 ± 0.3 s vs XGBoost, -0.76 ± 0.26 s vs `feat_gru` (the standard errors are optimistic by about sqrt(3) because the three seeds reuse the same recordings). Best epoch is ~11 of 25 for the feature models, so they overfit quickly.

### 6.5 E1: causal IIR filtering (`experiments/iir_filter`, 59 min)

Butterworth sections with `sosfilt`, state carried across windows (checked identical to whole-stream filtering), applied before the CNN and the features (artifact features stay on the raw signal). CV macro, 5 folds x 3 seeds, paired per recording against no filter:

| Filter | Macro | Difference |
|---|---|---|
| none | 42.2 | |
| low-pass 45 Hz | 41.5 | -0.57 +- 0.23 s |
| high-pass 0.1 + low-pass 45 | 41.8 | -0.19 +- 0.22 s |
| high-pass 0.3 + low-pass 45 | 41.9 | -0.24 +- 0.23 s |
| high-pass 0.5 + low-pass 45 | 42.1 | +0.09 +- 0.28 s |

Filtering is not a big lever; low-pass 45 Hz gives a tiny, borderline gain and is the sensible default (removes mains and the AF7 line noise). High-pass does nothing.

### 6.6 Distributional head (`experiments/dist_head`, 67 min), on the low-pass input

121-class head (0..600 s in 5 s steps), HL-Gauss soft targets (sigma 7.5 s), prediction = median.

| Variant | Macro (CV) | Difference vs point regression |
|---|---|---|
| point regression (`lp45`) | 41.5 +- 1.5 | |
| `dist_w`: cross-entropy weighted by the metric weights | **41.0 +- 1.2** | -0.61 +- 0.23 s (better on 55% of recordings) |
| `dist_u`: unweighted cross-entropy, weights applied only when decoding | 42.7 +- 1.9 | +1.88 +- 0.45 s |

A small consistent gain for `dist_w`; the 300-600 s bin is unchanged (MAE ~292 s), so the "far" region is limited by the EEG, not by the output representation. The metric weights must be used during training.

### 6.7 E8: bipolar channels and the 0.1-1 Hz band (`experiments/e8_bipolar`, ~80 min)

| Variant | Macro (CV) | Difference vs `lp45` |
|---|---|---|
| `e8a` bipolar channels (AF7-AF8, TP9-TP10) + 20 features | 41.9 | +0.30 +- 0.21 s |
| `e8b` `e8a` + 7 slow-band features | 42.5 | +0.85 +- 0.21 s |
| `e8c` 7 slow-band features only (correlation AF7-AF8 and TP9-TP10, band amplitudes) | 41.6 | +0.02 +- 0.19 s |

None helps; the slow-band features alone add exactly nothing. The "slow eye movement" idea is not supported in this form (per 5 s window, causal filter).

### 6.8 Public test split, seed 42 (`experiments/public_seed42`)

Train on the 500 public train recordings, score the 40 test recordings, one run per variant, seed 42, best epoch on inner-validation people drawn with the seed. Standard error of each figure ~4 s.

| Variant | W-bMAE | Mean of seeds 0/1/2 (for comparison) |
|---|---|---|
| `none` (unfiltered `cnn_feat_gru`) | 48.0 | 47.9 |
| `lp45` | 48.8 | 46.1 |
| `dist_w` | 48.0 | 45.5 |
| `dist_u` | 43.1 | 46.0 |
| `e8a` / `e8b` / `e8c` | 46.4 / 49.3 / 48.9 | 47.7 / 48.1 / 47.9 |

Seed 42 gives higher scores than seeds 0-2 for most variants (best epochs of 5-6 for `lp45` and `dist_w` show noisy checkpoint selection on ~30 inner-validation people). `dist_u` scores best here (43.1) although it is the worst in CV (+1.88 s over 693 recording-groups x 3 seeds): treat that as noise, not as a reason to choose it.

## 7. Conclusions

1. The 217 STFT/time-domain features carry most of the gain; the raw-waveform CNN adds under 1 s.
2. A GRU over per-window features beats hand-built EMA context fed to trees (-4.0 ± 0.4 s).
3. Elapsed time is not needed; models are not collapsing under the crop-start probe, so they do not appear to lean on the clock.
4. The 0-40 s bin (about half the metric weight) is still the weak spot for trees; the GRU models fix it only partly. A distributional head decoded with the metric-weighted median is the main untested idea.
5. The public 40-recording test has a standard error of ~4 s per model and only measures the seen group; trust the 540-recording CV over any public figure, and report the public number from seed 42.
6. Later experiments (6.5 to 6.7) added at most ~1 s in total: low-pass 45 Hz and the weighted distributional head help slightly; high-pass, bipolar channels and slow-band features do not.

## 8. Caveats and not verified

- XGBoost was refit on train plus inner validation; the GRU models were trained on 85% only, so the comparison is slightly unfair to the GRU.
- Each reported point is a single model per seed; no ensembling was tried.
- Nothing was run on the Codabench platform, and the sealed set's recording lengths, composition and cropping are unknown (private).
- The tuning of XGBoost, GRU and learning-rate schedules is minimal.
- Papers/models not verified in the report: DeepSurv details, EEGNeX, NeuroNet original, EEGPT original, mulEEG. EEGNeX/EEGInception are only listed as available in braindecode/neuralbench.
- The "tipping point" (Li et al. 2025) discussion in the report relies on a secondary source for the 0.82 ± 2.14 min figure.

## 9. Next steps

Decided by the user: **no solver is being written for now** ("just run the seed as the organizers do"). Open options:

1. Best configuration so far (CV macro ~41.0): `cnn_feat_gru` with low-pass 45 Hz input and the `dist_w` head; every gain after the 217 features is below ~1 s. Choosing a final recipe (and whether to train on all 540 recordings with a fixed epoch count and ensemble seeds) is still open.
2. The large remaining error sits in the far bin (300-600 s, MAE ~292 s, about a third of the score); representation changes did not help, so only new information (longer context, different features) could.
3. Untried items of `PREPROCESS_EXPERIMENTS.md`: E2 clipping/artifact features, E3 normalisation, E4 start-of-recording handling, E5 bad-channel flags, E6 recording-relative inputs, E7, E9, E10 Euclidean alignment.
4. If a submission is ever needed: a torch-only `submission.py` (numpy features, stored standardisation constants, GRU state kept across windows). The Codabench image has no xgboost.
5. Optional: finish the English version of the EDA report; add READMEs for `cnn_gru_feat`, `iir_filter`, `dist_head`, `e8_bipolar`.

## 10. Reproduce

```bash
# STFT + XGBoost (about 10 min, GPU), in tmux so it survives disconnects
tmux new-session -d -s stft_tree "bash experiments/stft_tree/run.sh; exec bash"
# CNN + GRU with raw EEG and 217 features (about 30 min, GPU)
tmux new-session -d -s cnn_gru_feat "bash experiments/cnn_gru_feat/run.sh; exec bash"
# quick plumbing check of either one
/root/miniconda3/bin/python experiments/cnn_gru_feat/run_cnn_gru_feat.py --smoke
```

Feature cache is built on first run (`experiments/stft_tree/cache`, ~30 s); results land in each experiment's `results/summary.md`, `scores_by_fold.csv` and `scores_by_recording.csv`.

## 11. Update 2026-10-17: E2, regime features, tree committee, ensembles

All measured; CV = 5 participant-grouped folds, macro of seen/unseen; public = 40 test recordings (SE ~4 s, run-to-run noise of one model ~2 s). The user's current goal is to push the **seed-42 public score** up (see memory `public-test-seed-42`); no benchopt solver is being written.

| Experiment | Finding |
|---|---|
| Regime features (18 per-time-to-N2-regime causal features) | no gain: GRU +0.59 +- 0.23 s (worse), HistGradientBoosting +0.10 +- 0.24 s |
| Tree committee (feature selection + all 8 scikit-learn tree families, `committee_stack` LAD combiner), `experiments/tree_committee` | CV 46.5 (equal to its best member and to XGBoost), public seed 42: 52.0; 5 s worse than the GRU; AdaBoost is broken in this setup (96 s) but the stacker ignores it; feature selection gave no clear benefit (K varies 30-381); selected features are stable (temporal-channel EMA features) |
| E2 clipping (`experiments/e2_clipping`) | clip at 400 / 780 uV: no effect (-0.12, +0.09 s); soft clip +1.06 s and 16 extra clip features +0.46 s are worse |
| **Seed ensembles of `lp45` + `dist_w` (`experiments/ensemble`)** | **CV 40.94 (single) -> 39.58 (5 models) -> 39.57 (8 models)**, -1.29 +- 0.08 s paired, better on 85% of recordings; saturates at ~5. Public, members seeded 42..: single 45.9 -> **5 models 43.9, 8 models 43.6** |
| Full-data training (100% of the training recordings, fixed 12 or 18 epochs) | single models -0.4 s in CV (40.53 vs 40.94) but ensembles are not better (39.89 / 39.64 vs 39.58) and the public score is worse (45.2 / 45.4 vs 43.9); a fixed 12-epoch schedule on 85% of the data equals the validation-selected epoch (41.03 vs 40.94) |
| GRU + tree blend | no gain: alpha = 1 (GRU only) is best, signed-error correlation 0.95; the earlier claim "trees are better in the 0-40 s bin" was wrong (GRU is better in all bins) |
| Pools of several recipes (10-20 models) | CV 39.4-39.6, public 43.6-45.1: nothing beyond the 5-8 member `val25` ensemble |

The same recipe re-run with seed 42 gave 48.0 first and 45.9 later (GPU non-determinism): single-run public numbers are only good to ~2 s. Best recipe now: ensemble of 5-8 `dist_w` GRUs on low-pass 45 Hz input, 85% of people per model with the epoch chosen on the other 15%. CV progression: 42.2 (`cnn_feat_gru`) -> 41.5 (low-pass) -> 41.0 (`dist_w`) -> 39.6 (ensemble). Details: `experiments/ensemble/results/summary.md` and `results/pools.md`.

**Update 2026-10-17 (later):** E3 normalisation (`experiments/e3_normalisation`): no normalisation helps (per-channel scale +0.12, InstanceNorm -0.32 +- 0.19, EMA 60/300 s +0.27/+0.20 s, all within noise or worse); keep the fixed 40 uV scale. E4 start-of-recording handling (`experiments/e4_start`): mask (1 s zeroed + first-window loss weight 0) -0.59 +- 0.15 s, random start 0-3 min -0.52 +- 0.20 s, both -0.69 +- 0.20 s against a same-session control (CV 41.09 -> 40.50); small but consistent in CV, no effect on the first-minute MAE (~220 s) or on the public score (47.4 for `both`, seed 42; noise +-2 s). Two submission zips exist in `tracks/sleep_onset/outputs/` (5 models 44.83, 8 models 44.19 on the public test, scored through benchopt); from now on experiments are scored offline (memory `offline-scoring-default`).
