# V1 evaluation review and precise v2 hypothesis

Source: user-provided `HUYLE_STQD_V1_EVALUATION_REPORT.md` dated 2026-11-05. This file summarizes that report critically; it does not claim to have re-run the server experiments or audited the original `huyle_run.log` (which was not provided in this turn).

## Source-grounded observations

- All 16 V1 configurations (8 variants × 2 seeds) completed SparrKULee pretraining and 5-fold LODO evidence. Only 2/16 completed real-switch evaluation, and only 1/8 completed external fusion.
- V1 synthetic-switch **classifier-only** mean LODO H: `stqd_full = 0.536` and `stqd_cosine = 0.588`. A no-time-bias variant was `0.533`; removing the time prior did not rescue the model in these synthetic LODO evaluations.
- One reported `stqd_full` real-switch evaluation: classifier-only AASD H 0.608 / Acc 49.5%, Carta H 0.549 / Acc 47.8%. These figures are *not* strong evidence of stimulus-conditioned attention decoding.
- The best reported external fusion AASD H 0.651 and Carta H 0.637 both occurred at classifier mixing weight `a=0.0`, which means **the classifier contributes nothing** to those results. Therefore it is incorrect to characterize AASD 0.651 as a classifier-induced improvement.
- The source reports termination by `SIGKILL (rc=-9)` while training a v1 all-five evaluation model. This is evidence of forced termination; the root cause (OOM, scheduler resource limit, user/server action, etc.) is not established by the provided report alone.

## What the report does *not* establish

- **True underfitting versus poor transfer:** mean LODO accuracy close to chance does not distinguish failure to learn from overfitting or data/task mismatch. To establish underfitting, inspect training BCE/accuracy trajectories, validation BCE, and held-out performance together. The source describes some training metrics but does not include a full epoch-level log.
- **Which component caused the failure:** the 8 v1 ablations indicate the cosine interaction did better than cross-attention, but do not isolate EEG encoder capacity, speech normalization, target-time readout, or pretraining-to-task mismatch as causes.
- **Challenge compliance:** a bounded input window and passing a model-only test are not equivalent to a full end-to-end perturbation/leakage test through preprocessing, feature extraction, fusion and expansion.
- **A matched baseline improvement:** comparing a single v1 classifier with a four-classifier ensemble, or a 4-decoder+2-classifier fusion with a 4+4 fusion, is not a controlled attribution experiment.

## V2 hypothesis and pre-registered comparison

**H:** The compact v1 EEG TemporalBlock encoder may fail to extract sufficiently useful auditory-attention information; changing *only the EEG encoder* to the original weight-shared VLAAI backbone may improve the new joint interaction model's classification evidence under unchanged acoustic processing, readout and source-domain evaluation.

This is a **hypothesis**, not a validated mechanism. V2 changes `EEGEncoder` only: spatial/channel projection and 3 repeated shared VLAAI blocks output `[B,320,64]`, then the same stride-4 pooling produces `[B,80,64]`. Speech and scorer modules remain as before. The `v2_cosine` ablation addresses whether the previous stronger, simpler similarity head is still preferable after increasing EEG encoder representational capacity.

## Interpretation of future v2 results

1. Compare v2 single seed 0 **classifier-only** LODO H, Acc, SDT, and per-domain H to v1 seed 0 classifier results and the original VLAAI classifier seed-0 result, with clearly distinguished training configurations.
2. Compare `v2_full` vs `v2_cosine` with all other architecture and training choices identical. Differences should be interpreted in the context of model capacity and optimizer convergence.
3. For AASD/Carta, report **classifier-only** scores and 4-decoder+1-classifier fusion with fixed source-selected parameters. Do not present grid-maximized test scores as honest independent tests.
4. A winning v2 one-seed configuration remains a preliminary engineering candidate until seed variance, model selection and full timing/compliance are evaluated.

## Why no normalization or STFT changes yet?

V1 proposed that per-window normalization might remove useful dominance information. That remains plausible, but the v1 report does not empirically establish it. Combining a `dom` normalization change and a backbone replacement in v2 would confound their effects, so v2 intentionally preserves the earlier speech pathway. Any future feature-processing experiment should be separately tagged and evaluated.
