---
name: real-switch-findings
description: "Results 2026-10-23..28 on REAL switches (AASD, Carta) vs synthetic LODO - A1/A2/A3 no gain, 4-s evidence lag, HMM recalibration, schedule noise, stem_ms96 audit; where jobs and results live"
metadata:
  node_type: memory
  type: project
  originSessionId: d4dac82a-f3e4-4196-a274-a5c105cb0541
  modified: 2026-10-28T11:07:11.334Z
---

**Real-switch data in the project:** AASD (18 subj, 900 trials x 60 s, spontaneous, key-press labels, ~5.1 switches/trial, median dwell 10 s) and Carta2026 (24 subj, 480 x 175 s, cued, 6 switches/trial, dwell ~25 s). KUL-AV-GC only has a DIRECTION switch at 300 s (same talker moves side): invisible to a speaker decoder. Labels are hard steps; the brain response ramps over seconds.

**Measured (2026-10-28):**
- Evidence lags the label by about 4 s on real data (per-subject median +4 s; Acc 0.538 -> 0.559 AASD, 0.598 -> 0.619 Carta when labels are shifted), ~0 s on synthetic switches. Not removable at scoring (label at t may use EEG only to t+1 s); it is part of SDT (13 s AASD, 18 s Carta).
- Evidence autocorrelation 0.92-0.95 at 0.25 s, but tempering the HMM likelihood (scale > 1) HURTS; lower p_stay (0.9) gives only +0.003..0.012 H (`results/research/hmm_real_crossfit.csv`). Mean-of-4-logit baseline on real: AASD H 0.633, Carta 0.643 (p 0.998).
- **A1 (train on the other real set), A2 (real-window share 25/50/100 %, straddle windows, no fake switch on real), A3 (labels shifted 2/4 s)**: none beats the baseline without real data (AASD 0.621-0.630 vs 0.633; Carta 0.594-0.636 vs 0.646); Acc unchanged (52 % / 58 %); r100 hurts Carta (0.594). Bottleneck = base accuracy of the EEG, not the switch dynamics. A3 lag4s, A4 (aux "switch in last 5 s" head + gated HMM, `hmm_gated.py`) and `5500_lodo_xreal_s0/s1` (LODO with AASD+Carta in training) were still queued/running at the time of writing; results in `results/research/ext_clf_*_a1h*/a2*/a3*/a4*_s*.json`, `logs/a{2,3,4}_*`.
- **Schedule noise** (5 switch schedules, `results/research/schedule_noise.csv`): best fusion H 0.7975 +- 0.0015 (range 0.7962-0.7996); per dataset +-0.005..0.01. Retraining is bit-exact (H 0.789 identical). Differences < ~0.003 in LODO are noise.
- **eeg-thao.zip `stem_ms96`** (extracted in `eeg-thao_unzipped/`, never run its code, `.pkl` from it not unpickled): synthetic LODO 0.863 (5 schedules 0.854 +- 0.005) but its first 24 s are a constant "column 0" (99.9 % vs 0.3 % after a column swap); symmetrised version 0.827; on real switches AASD 0.613-0.618, Carta 0.546-0.561 (< mean4). Tier-1 code in the zip is identical to ours; only its tier-2 net and the model selection after seeing outer scores are the problem.

**How to apply:** do not tune further on synthetic LODO alone; judge every idea on the two real folds AND on LODO (paired by seed). Plan and experiment list: `REAL_SWITCH_STRATEGY.md` in the repo root. Code: `eval_external_clf.py --train-ext/--aug-set clf_real_*`, `direct_classifier.py --extra-real`, `clf_reschedule.py`, `schedule_noise_score.py`, `eval_stem_external.py`. See [[organiser-reply-no-structure]], [[design-priorities]], [[decoder-lodo-findings]].

**Decisions (user, 2026-10-28): ideas that are not good are not run again; they are only recorded.**
- STOPPED, do not re-run: **A1** (train on the other real set: real AASD 0.626 vs 0.633, Carta 0.628 vs 0.646; LODO synthetic with AASD+Carta in training, classifier alone, `5500_lodo_xreal_*`: seed 0 0.7706 vs 0.7887, seed 1 0.7616 vs 0.7761 = -0.015..-0.018, AVGC -0.026/-0.012); **A2** (real-window share 25/50/100 %, straddle, no fake switch: all below baseline, r100 Carta 0.594); **A3** (labels shifted 2 s / 4 s / 4 s soft: AASD 0.628-0.630, Carta 0.628-0.636, below 0.633 / 0.646); **stem_ms96** (zip); HMM likelihood tempering (scale > 1).
- STILL RUNNING to decide: **A4** (aux "switch in last 5 s" head + gated HMM; with real data in training the gate adds +0.007..0.014 over no gate but the net is weaker, final AASD 0.636 / Carta 0.643 ~ baseline; base-recipe runs `5600_lodo_a4_s*` + `5601_ext_a4base_s*` pending) and **A5** (decoder with real data: `57xx`, `58xx` jobs pending). If their results are not good they are stopped and recorded the same way.
- **KEPT (user, 2026-10-29): B1 semi-Markov HMM** (`hmm_semimarkov.py`, `results/research/semimarkov.csv/.txt`). Baseline HMM 0.998 on the best fusion: AASD 0.638, Carta 0.639, LODO synthetic 0.798. `m5 k1.5`: real +0.016/+0.012, synthetic -0.016; heavy tail `m10 k0.6`: +0.007/+0.008, -0.005; `m20 k0.6`: +0.005/+0.005, -0.002. The gain comes from a short mean dwell (geometric `m5 k1` already +0.011/+0.010, synthetic -0.008), not from the shape; real wants fast switching, synthetic slow. Parameters NOT fixed: do not hard-code m = 5 s (it is a fit to the real sets' switch rate, not the real dwell 10-25 s). Leakage / column-swap checks on it not yet run.
- **STOPPED (user, 2026-10-29): B2 BOCPD** (`hmm_bocpd.py`, `results/research/bocpd.csv/.txt`; variants `learned`/`robust` in `hmm_bocpd_followups.py`, `bocpd_*_followup_v2.csv`). Real +0.003..+0.012, synthetic -0.008..-0.011 vs HMM 0.998; plain HMM p_stay 0.95 gives AASD 0.649, Carta 0.648, synthetic 0.7903 (AVGC 0.713 vs 0.726), so BOCPD is no better; `bma` (no hazard choice) does not adapt to the slow synthetic regime. The gain over the baseline comes from a lower confidence ceiling (high hazard / low p_stay), not from the Bayesian filter. Decision rules are near their ceiling (~+0.01): effort should go to evidence quality (new views, E-list). Do not re-run.
- **Check 2026-10-29** (`leakage_test_b1.py`, `results/research/leakage_b1.txt`, seed 0, decoder = SparrKULee-pretrained weights, classifier = real fold weights): best pipeline + HMM/B1 uses <= 0.969 s lookahead, negative controls flagged, column swap flips exactly, EEG with another trial's stimuli shows no column-0 / position prior (max share 0.545). Not run for BOCPD.
- **STOPPED (user, 2026-10-30): C2 label delay** (`clf_label_delay_max_s=2`: label flips first, the stimulus-EEG swap follows by U(0, 2 s); classifier alone + HMM, 2 seeds; `results/research/c2delay_pstay_lodo_s*.csv`, `ext_clf_*_w10b3c2delayP_s*.json`). Real vs sw2 same seed: AASD +0.001..+0.007, Carta -0.004..-0.011 (SDT longer); synthetic LODO seeds disagree (-0.009 / +0.008 at p 0.998): seed-to-seed spread (~0.017) exceeds the effect. Not run as an ensemble. Do not re-run.
- **C1 cross-fade ensemble 4+4 (finished 2026-10-29, decision pending)**: vs matched sw2 at p 0.998: AASD 0.642 vs 0.638, Carta 0.646 vs 0.639, synthetic LODO 0.7742 vs 0.7981 (-0.024; AVGC 0.713 vs 0.726) -> fails the section-9 criteria. **B3 learned rule was KEPT by the user (2026-10-30)**: V1 collapses on synthetic/Carta, V2 +0.013 synthetic but no real gain; keep researching, not chosen for submission.
- Evaluation gaps listed 2026-10-30 (no automated acceptance report, no subject-level CIs, only two real sets, one synthetic generator, no oracle headroom, checks only run for B1): all points lie on one real<->synthetic frontier of the HMM p_stay.
- **STOPPED (user, 2026-10-31): C4 babble in the envelope** (`clf_p_babble`; classifier alone, 2 seeds): AASD +0.001..+0.003, Carta -0.007..-0.015, synthetic -0.004..+0.010, AVGC mostly down. Do not re-run. **C3 attention lapse** finished, decision deferred by the user: AASD +0.002..+0.006, Carta down (`c3soft` -0.007..-0.017). C1-C4 all show AASD slightly up, Carta down.
- **STOPPED (user, 2026-11-02): C6 time warp** (`clf_p_warp`; classifier alone LODO +0.003..+0.011 on seeds 0-1, but the 4+4 ensemble equals sw2: 0.7973 / 0.6367 / 0.6361 vs 0.7981 / 0.6380 / 0.6388 at p 0.998; seed pairs disagree on real) and **C7 real-switch straddling windows** (`--cv-k 3`; LODO -0.021..-0.032, Carta -0.027..-0.082, AASD mixed). Do not re-run. C3 and C5 finished, decision pending.
- **Seed noise on real with 2 seeds is about +-0.005..+-0.014** (C6 seed pairs flip AASD/Carta signs): never read 0.005 differences from two classifier seeds; the "AASD up, Carta down" pattern of C2-C5 may be partly noise.
- **Stimulus leak trap:** in Carta2026 the same T has an identical label sequence for all 24 subjects, so training on Carta needs a split by subject AND stimulus (`eval_external_clf.cv_split`). AASD labels differ between subjects (agreement 0.53).
- **Foundation models, 2026-11-02** (weights in `experiments/external_data/foundation_models/`, code `fm_input.py`, `fm_embed.py`, `fm_screen.py`): CBraMod embedding, linear read-out d' 0.09-0.12 LODO, 0.006 AASD, 0.063 Carta, weaker than plain 4-Hz EEG (0.15-0.29 / 0.012 / 0.089), classifier 0.47-0.71 / 0.084 / 0.244, corr with classifier 0.07-0.11, best mix +1.2 %. **D2 (LoRA) and D3 (scratch) HELD in `runners/jobs/held/` by the user** until a screen passes (above null >= 3 SE on a real set and LODO AND better than plain EEG or mix gain >= 5 %). D1 (`7510`, `7511`) and screens `7600`-`7602` (no PCA, per electrode, BrainOmni) were running. Lesson: a waiting job (`until [ -f ... ]`) holds a queue slot; check the preparation job has passed its first steps before ending the turn.
- **Foundation backbones 2026-11-03**: column averaging made A/B interchangeable (no learning) -> fixed with `ColumnReadout`; fine-tuning from step 0 stays at chance (any lr, dropout on/off); only frozen / LP-FT learn; classifier collapses when the encoder is unfrozen. Decoder CBraMod LP-FT LODO ~= or below `orgbase`, ~15x slower. `orgbase` (no SparrKULee, no Fuglsang): decoder LODO 0.748 / 0.756, classifier 0.763 / 0.763. **H1 (stimulus pretraining) STOPPED by the user before results**; H5 (`fm_front`, frozen FM on EEG only before vlaai_ws) being screened. Always run a memorise-16-windows test before queueing a new architecture; when stopping a job kill the python child, check with `ps`.
- **2026-11-03..07 (old-metric H unless noted; full tables in `REAL_SWITCH_STRATEGY.md` §10):** H5 frozen-FM front: no gain for any of 5 FMs (BIOT ~0 even with raw EEG = likely wiring bug, unverified). CBraMod LP-FT decoder below `orgbase` on LODO (0.741 vs 0.747/0.756) and Carta (0.61 vs 0.62-0.63). H1 pretrain done for 4 folds + ALL, USTC + H1 eval jobs never ran (default scheduler stopped ~13:00 UTC 03/11; only the `nested` queue runs). Nested antisymmetric stage-2 Transformer over 16 evidence streams: alone raises AVGC/LODO but lowers real; 25 % mix with 4+4 sw2: LODO 0.804-0.805, real ~= baseline. Recipe aug 4+4: LODO -0.004, real noise. **BMMS3** (BMMSNet adaptation + EEG enhancement + env/onset/dom + SparrKULee pretrain): single classifier vs sw2 same seed raises real (Carta Acc +2 pts, H 0.645/0.659 vs 0.640/0.643), LODO -0.011/+0.000; 0.25 in the ensemble: Carta new-metric 0.297 -> 0.315. **Stimulus-only control (EEG = 0) scores AASD ~0.61 / LODO ~0.53 under the old metric**: the old metric gave free H at chance Acc. No user keep/drop decision yet for stage-2 Transformer, recipe, BMMS3. See [[metric-kappa]].
- **DROPPED (user, 2026-11-07): eye-artifact filtering** (virtual-EOG regression, SSP, ICA, ASR, blink augmentation; list in `ACC_EXPERIMENT_PLAN.md`): not to be run; frontal-channel ablation already showed no reliance on eye signals (±0.004).
- **Decisions (user, 2026-11-08): BMMS3 KEPT in the main pipeline** (4 dec + 4 DirectNet sw2 + 4 BMMS3, equal thirds `eq3`, new metric p 0.998: LODO Acc 73.40 / H 0.619, AASD 52.77 / 0.104, Carta 59.84 / 0.322 vs ref 73.35 / 0.621, 52.81 / 0.106, 59.24 / 0.306; end-to-end leakage test for BMMS3 still to run). **STOPPED, do not re-run: mirror augmentation** (`p_mirror`, AVGC -1.6, Carta -0.8, LODO -0.5 Acc) and **lag jitter ±1 sample** (`clf_lag_jitter=1`, real +0.15..0.35, LODO -0.2..-0.4 Acc). AVGC is ~31 ms misaligned (optimum shift -1 sample, DTU 0); a global shift is ruled out.
