---
name: blind-test-sets
description: Official Terms and test-phase rules (read 2026-10-05 from Codabench competition 17856): only the trial under test may be used, no IDs, within-trial causal adaptation allowed, 2 submissions, dates; consequence = prep mat_rea (no EA carry)
metadata:
  type: project
---

Read on 2026-10-05 from the Codabench API (`https://www.codabench.org/api/competitions/17856/` returns JSON with `terms` and `pages`; the normal page is a JS app and cannot be fetched). Local copy of the text: `docs_challenge/*.txt` in the repo root.

**Blind test sets** (both "will contain switches in attention"; real vs synthetic is NOT stated):
- KUL-DLAAD: 64 ch BioSemi, Dutch, eye gaze controlled, 19 subjects, 8 trials/subject, ~50 min/subject (trials ~6 min).
- NJU-SUSTech-sAAD-GC (new): 64 ch NeuroScan, Chinese, eye gaze controlled, 12 subjects, ~60 min/subject.
- Raw + preprocessed EEG and envelopes are released, no labels. Labels to submit: per sample at 128 Hz for EVERY test trial; either attended-speaker (envelope 1 vs 2) or attended-direction labels.

**Test-phase rules (Terms 6.3, 6.4, 9.1, Timeline page):**
- **Only data from the specific test trial may be used at inference; no data from other test trials or subjects** (6.3). Label at t may use samples [0, t + 1000 ms] of the same trial, all pipeline lags counted.
- No fine-tuning / unsupervised / self-supervised update of model parameters with test data, **except within-trial and causal**.
- No dataset / subject / trial ID in any form; the decoder takes EEG and (optionally) envelopes only.
- No different architecture / weights for different test samples; a mixture of experts is allowed only if experts are selected automatically from the (causal) input.
- Exempt from the lookahead, each once per pipeline: filtfilt (same order), removeBadChannel, repairBadSegment, full-trial z-score; the supplied preprocessed files may be used as they are.
- No restriction on the training procedure; external data allowed (respect licences).
- Code (training + inference) must be submitted; compliance is checked for at least the top 5; results must be reproducible.

**Schedule:** development phase until 2026-11-08 23:55 UTC (a format-check submission on KUL-AV-GC opens in its last two weeks); test phase 2026-11-09 to 2026-12-08 23:55 UTC; **at most 2 test submissions, the LATEST counts (not the best)**; results 2026-12-14. Top 5 are invited to a 2-page ICASSP paper.

**Scoring:** Acc = mean over subjects, test sets weighted equally; L = per subject the MEDIAN of exp(-SDT/60) over its switches (undetected = 0; search window 100 s, sustain 0.5 s), mean over subjects; H = harmonic mean of Acc and L.

**Why it matters / how to apply:**
- Prep `mat_reac` carries the Euclidean-alignment state across a subject's trials -> **not allowed at test** (6.3). Use `mat_rea` (EA restarts in every trial; `--prep mat_rea` in run_experiment.py and direct_classifier.py; Fuglsang cache `mat_rea` built 2026-10-05). Carry was worth +0.005 to +0.010 H for one decoder on the corpus (short trials); test trials are long. No-carry runs of the best config: jobs 381-388, `logs/fuse_nocarry.log`, `logs/fuse_nocarry_5_10.log`.
- Cross-trial per-subject adaptation is forbidden; within-trial causal adaptation (even of parameters) is allowed.
- The user wants the submission to contain decoders + 5-s AND 10-s classifiers (2026-10-05).
- With only 2 submissions and "latest counts", choose robust settings (p_stay 0.998, not 0.9997: worst value on real switches).

See [[zscore-allowed]], [[decoder-lodo-findings]].
