# results_stem_alibi

ALiBi causal attention bias on `stem_ms96`: remove absolute `self.pos`, inject head-specific ALiBi slopes into the **full** causal attention mask of the stem LogitTransformer (2 layers, d=64, nhead=4, window 96, ms features, dilated residual stem kept). Slopes fixed: `[0.25, 0.0625, 0.015625, 0.00390625]`. Not v3/v4 local relative-bias span 6.25s; not RoPE.

Init from `models/stem_ms96_swa/{held-*,all5}.pt` (drop `pos`; restore causal `-inf` mask after SWA buffer avg). Fine-tune 1000 steps, batch 512, lr 8e-4, AdamW, cosine, EMA 0.995. Early check @80 steps on held-AVGC: mean loss **0.2541** (kept). HMM preset p_stay 0.994 scale 0.8. Also scored causal tau-30 high-pass of ALiBi evidence (`hp30`; corpus emissions). Never used AASD/Carta labels.

Code: `stem_ms96_alibi.py`. Checkpoints: `models/stem_ms96_alibi/`. JSON: `results/research/stem_ms96_alibi.json`. Log: `experiments/stem_ms96_alibi.log` / `stem_ms96_alibi_console.log`.

## LODO (preset 0.994 / 0.8)

| Held-out | ALiBi H | ALiBi Acc % | SWA H/Acc | IRM↔SWA fuse H/Acc | hp30 H | hp30 Acc % |
|---|---:|---:|---|---|---:|---:|
| AVGC | 0.7654 | 69.15 | 0.8015/74.91 | 0.8100/75.14 | 0.7053 | 65.85 |
| DTU | 0.8648 | 77.16 | 0.9003/89.52 | — | 0.8054 | 67.71 |
| KUL | 0.8443 | 75.77 | 0.8769/82.35 | — | 0.8051 | 69.45 |
| UHD | 0.8117 | 74.68 | 0.8553/80.09 | 0.8580 | 0.7728 | 70.41 |
| USTC | 0.8737 | 78.42 | 0.9161/86.19 | — | 0.8307 | 71.42 |
| **mean** | **0.8320** | **75.04** | **0.8700/82.61** | **0.8725/82.58** | | |

## External (preset)

| Set | ALiBi H | ALiBi Acc % | linear_hp30 baseline | ALiBi_hp30 H/Acc |
|---|---:|---:|---|---|
| AASD | 0.6308 | 51.53 | 0.6422 / 53.12 | 0.6273 / 51.34 |
| Carta2026 | 0.6384 | 58.11 | 0.6520 / 60.54 | 0.6452 / 59.10 |

## Targets

| Target | Result |
|---|---|
| Mean H≥0.85 Acc≥78 | miss (0.8320 / 75.04); below SWA/fuse |
| AVGC H≥0.85 Acc≥78 | miss (0.7654 / 69.15); **regressed** vs fuse 0.8100/75.14 |
| UHD H≥0.85 | miss (0.8117); lost SWA/fuse bar |
| AASD Acc>60 | miss (51.53); flat vs stem family, below linear_hp30 |
| Carta Acc>60 | miss under ALiBi (58.11); hp30 59.10 still short of linear_hp30 60.54 |

**No open target newly met.** Replacing absolute pos with full-attention ALiBi on this stem hurts LODO (especially AVGC/UHD) and does not help AASD Acc.

## NEXT IDEA (coordinator)

AVGC still open (~−0.04 H / −2.86 Acc vs fuse). Full-window ALiBi on the logit stem appears to discard useful absolute-pos adaptation from SWA without adding switch robustness. Prefer **non-attention** AVGC levers still unused (e.g. subject-embedding FiLM if absent from ALREADY_TRIED) or honest fusion of complementary streams; avoid another relative/ALiBi attention rewrite of stem_ms96.
