# results_stem_rope

RoPE on `stem_ms96` attention **Q/K**: remove absolute `self.pos`, apply rotary embeddings inside a custom causal stem encoder (2 layers, d=64, nhead=4, window 96, ms features, dilated residual stem kept). Causal `-inf` mask kept. Not ALiBi; not v3/v4 local relative-bias span 6.25s; not env_xattn.

Init from `models/stem_ms96_swa/{held-*,all5}.pt` (drop `pos`; remap MHA → RoPE qkv/out_proj; restore causal mask after SWA buffer avg). Fine-tune 1000 steps, batch 512, lr 8e-4, AdamW, cosine, EMA 0.995. Early check @80 steps on held-AVGC: mean loss **0.2546** (kept). HMM preset p_stay 0.994 scale 0.8. Also scored causal tau-30 high-pass of RoPE evidence (`hp30`; corpus emissions). Never used AASD/Carta labels.

Code: `stem_ms96_rope.py`. Checkpoints: `models/stem_ms96_rope/`. JSON: `results/research/stem_ms96_rope.json`. Log: `experiments/stem_ms96_rope.log` / `stem_ms96_rope_console.log`.

## LODO (preset 0.994 / 0.8)

| Held-out | RoPE H | RoPE Acc % | SWA H/Acc | ALiBi H/Acc | IRM↔SWA fuse H/Acc | hp30 H | hp30 Acc % |
|---|---:|---:|---|---|---|---:|---:|
| AVGC | 0.8124 | 75.66 | 0.8015/74.91 | 0.7654/69.15 | 0.8100/75.14 | 0.7784 | 70.99 |
| DTU | 0.8969 | 89.15 | 0.9003/89.52 | 0.8648/77.16 | — | 0.8250 | 70.64 |
| KUL | 0.8678 | 81.88 | 0.8769/82.35 | 0.8443/75.77 | — | 0.8465 | 74.44 |
| UHD | 0.8541 | 80.11 | 0.8553/80.09 | 0.8117/74.68 | 0.8580 | 0.8147 | 73.09 |
| USTC | 0.9245 | 86.59 | 0.9161/86.19 | 0.8737/78.42 | — | 0.8322 | 71.28 |
| **mean** | **0.8711** | **82.68** | **0.8700/82.61** | **0.8320/75.04** | **0.8725/82.58** | | |

## External (preset)

| Set | RoPE H | RoPE Acc % | linear_hp30 baseline | RoPE_hp30 H/Acc |
|---|---:|---:|---|---|
| AASD | 0.6216 | 51.04 | 0.6422 / 53.12 | 0.6049 / 51.04 |
| Carta2026 | 0.5392 | 48.06 | 0.6520 / 60.54 | 0.5633 / 49.66 |

## Targets

| Target | Result |
|---|---|
| Mean H≥0.85 Acc≥78 | hit numbers vs ALiBi, but ≈SWA/fuse (0.8711 / 82.68); no new bar |
| AVGC H≥0.85 Acc≥78 | miss (0.8124 / 75.66); tiny lift vs fuse 0.8100/75.14 and SWA 0.8015/74.91 |
| UHD H≥0.85 | keep (0.8541); ≈SWA 0.8553 |
| AASD Acc>60 | miss (51.04); Δ−2.08 vs linear_hp30 53.12 |
| Carta Acc>60 | miss (48.06); far below linear_hp30 60.54 |

**No open target newly met.** RoPE recovers the ALiBi LODO collapse and lands ≈SWA / IRM↔SWA fuse; AVGC only +0.002–0.01 H. External Acc flat in the stem/SWA family (~51 AASD).

## vs ALiBi / SWA (headline)

| Method | LODO mean H/Acc | AVGC H/Acc | AASD Acc | Carta Acc |
|---|---|---|---:|---:|
| ALiBi | 0.8320 / 75.04 | 0.7654 / 69.15 | 51.53 | 58.11 |
| SWA | 0.8700 / 82.61 | 0.8015 / 74.91 | ~51 | ~48 |
| RoPE (this) | **0.8711 / 82.68** | **0.8124 / 75.66** | 51.04 | 48.06 |
| IRM↔SWA fuse | 0.8725 / 82.58 | 0.8100 / 75.14 | 50.94 | ~48 |

## NEXT IDEA (coordinator)

AVGC still open (~−0.037 H / −2.3 Acc vs 0.85/78). Abs-pos→attention rewrites are saturated: ALiBi hurt; RoPE ≈SWA. Prefer **non-attention** AVGC/AASD levers still unused (e.g. subject-embedding FiLM if absent from ALREADY_TRIED) or honest fusion of complementary streams that already move Acc; do not rerun RoPE / ALiBi / another stem attention pos swap.
