# HUYLE STQD v1 Framework — Complete Evaluation Report

**Date:** 2026-11-05  
**Evaluator:** Research Colleague (Agent)  
**Standard:** A/A* Conference Review Rigor  
**Purpose:** Archive for independent re-evaluation by another agent

---

## 1. Executive Summary

The **huyle** directory implements **STQD v1** (Stimulus-conditioned Temporal Query Decoder), a clean-slate replacement for the `direct_classifier` in `decoder_lodo`. It retains the original 5-second reconstruction decoder (`vlaai_ws`), data splits, LODO protocol, evidence/HMM pipeline, and official scorer — changing only the 10-second direct classifier architecture.

**Bottom line:** The framework is **well-engineered and compliant**, but the STQD classifier **significantly underperforms** the original VLAAI-based classifier on synthetic-switch LODO (0.54 vs 0.75 H). Fusion with the original decoder partially recovers performance on AASD real switches but still trails on Carta. The run was **interrupted** (SIGKILL) before completing all variant × seed evaluations on real-switch data.

---

## 2. Framework Specification

### 2.1 Design Intent (from README.md)
> Replace only the *direct classifier* with a stimulus-conditioned temporal query decoder (STQD), without changing the existing 5-second reconstruction decoder, data splits, original evaluation scorer, or downstream evidence/HMM protocol.

### 2.2 STQD v1 Architecture (`stqd_model.py`)

| Component | Specification |
|-----------|---------------|
| **Input** | EEG `[B, 320, 58]` (10s @ 32Hz), Speech `[B, 320, 2, 3]` (env, onset, dom) |
| **EEG Encoder** | 1×1 Conv(58→64) → 3× TemporalBlock(dilations 1,4,16) → AvgPool(stride=4) → `[B, 80, 64]` |
| **Speech Encoder** | Separate stems: env/onset 24ch, dom 16ch → shared TemporalBlocks → AvgPool(4) → `[B, 2, 80, 64]` |
| **Fusion** | 4-head Cross-Attention: queries=speech, keys/values=EEG + soft temporal bias (μ=+0.20s, σ=0.55s) |
| **Readout** | Fixed token 72 (288//4 = 2.25s before window end) → Shared head → `logit = score[0] - score[1]` |
| **Antisymmetry** | Exact: same encoder/head for both candidates; column swap flips logit sign |

### 2.3 Variants (8 total)

| Variant | Change vs `stqd_full` |
|---------|----------------------|
| `stqd_full` | All features, separate stems, cross-attn, time bias |
| `stqd_joint_stem` | Concatenate env/onset/dom → joint conv |
| `stqd_env` | Envelope only |
| `stqd_env_onset` | Envelope + onset |
| `stqd_env_dom` | Envelope + dominance |
| `stqd_no_time_bias` | Remove cross-attn temporal bias |
| `stqd_global_query` | Pool speech to single query per candidate |
| `stqd_cosine` | Cosine similarity (projected) instead of cross-attn |

### 2.4 Training Protocol

| Phase | Data | Objective | Epochs |
|-------|------|-----------|--------|
| **SparrKULee Pretrain** | Single-speaker (match vs shifted-envelope mismatch) | BCE | 80 (min 25 no-ES) |
| **LODO Fine-tune** | 4/5 corpus datasets + Fuglsang2020, synthetic switches (p=0.5, p_sw2=0.5) | BCE | 40 (min 10 no-ES) |
| **All-5 Train** | All 5 corpus + Fuglsang, same augmentations | BCE | 40 |
| **Real-switch Eval** | AASD (18 subj, 900 trials), Carta2026 (24 subj, 480 trials) | HMM (p_stay sweep) | — |

**Augmentations:** EEG cluster drop (p=0.3, 3-8 ch), feature drop (p=0.2), synthetic switch (50%), double-switch (50%)

---

## 3. Execution Status

```
Total planned: 8 variants × 2 seeds = 16 classifier configurations
├── SparrKULee Pretraining:     16/16  ✅ COMPLETE
├── LODO 5-fold (synthetic):    16/16  ✅ COMPLETE
├── Real-switch eval (AASD+Carta): 2/16  ⚠️ PARTIAL (stqd_full seeds 0,1 only)
├── External Fusion:             1/8   ⚠️ PARTIAL (stqd_full only)
└── Decoder LODO Fusion:         8/8   ✅ COMPLETE (fuse_general.csv)
```

**Failure:** `SIGKILL (rc=-9)` on 2026-11-03 09:44 during `stqd_joint_stem` seed=0 real-switch training (epoch 22). Likely OOM or server timeout.

**Log location:** `logs/huyle_run.log` (4094 lines, complete stdout/stderr)

---

## 4. Results Summary

### 4.1 Classifier-Only LODO (Synthetic Switches) — Mean over 5 held-out datasets

| Variant | Seed 0 H | Seed 1 H | **Mean H** | Acc | SDT | Δ vs Baseline (0.75) |
|---------|----------|----------|------------|-----|-----|---------------------|
| `stqd_cosine` | **0.592** | **0.585** | **0.588** | 55.3% | 26.9s | **−0.162** |
| `stqd_full` | 0.538 | 0.534 | 0.536 | 50.6% | 32.8s | −0.214 |
| `stqd_env_onset` | 0.544 | 0.521 | 0.533 | 50.4% | 34.7s | −0.217 |
| `stqd_no_time_bias` | 0.534 | 0.532 | 0.533 | 50.3% | 32.4s | −0.217 |
| `stqd_env_dom` | 0.527 | 0.514 | 0.521 | 50.1% | 32.6s | −0.229 |
| `stqd_env` | 0.518 | 0.517 | 0.518 | 50.0% | 35.9s | −0.232 |
| `stqd_global_query` | 0.530 | 0.513 | 0.522 | 50.3% | 34.6s | −0.228 |
| `stqd_joint_stem` | 0.504 | 0.502 | 0.503 | 49.7% | ∞/38.7s | −0.247 |

> **Key observation:** Even the best variant (`stqd_cosine`) trails the original `direct_clf` (~0.75 H) by **0.16 H**. The classifier is the bottleneck.

### 4.2 Real-Switch Evaluation — `stqd_full` Only (Seeds 0, 1)

#### Classifier Only (p_stay sweep)

| Dataset | Best H | Best Acc | Best p_stay | SNR₁ |
|---------|--------|----------|-------------|------|
| **AASD** | 0.608 | 49.5% | 0.9 | 0.008 |
| **Carta2026** | 0.549 | 47.8% | 0.95 | 0.042 |

#### Fusion: 4 Decoder + 2 Classifier (a = classifier weight)

| Dataset | Best H (a=0.0) | Best Acc | p_stay | Notes |
|---------|----------------|----------|--------|-------|
| **AASD** | **0.651** | 53.7% | 0.95 | +0.01 vs decoder-only baseline (~0.64) |
| **Carta2026** | **0.637** | 58.3% | 0.99 | −0.003 vs decoder-only baseline (~0.64) |

> **Critical:** Fusion uses **4 decoders + 2 classifiers** vs baseline's **4 decoders + 4 classifiers**. Not a fair comparison. Classifier weakness is masked by decoder dominance.

### 4.3 Decoder LODO Fusion (`fuse_general.csv`) — All Variants Complete

All 8 variants fused with 4 decoder seeds (`vlaai_ws__eaP_n5_t60_aug_cluster_onset_dom_xFUG_ptSPK` s0-s3). Results consistent: STQD variants add marginal value over decoder-only; `stqd_cosine` strongest.

---

## 5. Architectural Analysis

### 5.1 Strengths ✅

| Aspect | Evidence |
|--------|----------|
| **Column-swap antisymmetry** | Exact by construction; verified in unit tests (`tests/`) |
| **No forbidden inputs** | No dataset/subject/trial IDs, no position encoding — compliant |
| **Causal by design** | Window construction enforces ≤0.75s lookahead; `leakage_test.py` compatible |
| **Clean inductive bias** | Separate EEG/speech encoders → cross-attention retrieval; cleaner than concatenated VLAAI |
| **Resumable pipeline** | Fold-level checkpoints, seed-controlled, single consolidated log |
| **Controlled ablations** | 8 variants isolate stems, features, fusion, query type, temporal bias |

### 5.2 Weaknesses ❌

| Issue | Severity | Evidence |
|-------|----------|----------|
| **Severe underfitting on synthetic LODO** | Critical | 0.54 H vs 0.75 H baseline (−28% relative) |
| **Temporal bias unvalidated** | High | "NOT a proven physiology model" (README); fixed μ=0.20s, σ=0.55s |
| **Fixed readout token (72)** | High | Hardcoded 2.25s before window end; no per-dataset adaptation |
| **Single-scale EEG encoding** | Medium | Stride-4 pooling only; no multi-resolution (cf. VLAAI, Deformer) |
| **Pretraining→fine-tune gap** | Medium | SparrKULee single-speaker match/mismatch → synthetic instant-switch LODO |
| **Per-window per-candidate z-score** | Medium | Line 165 `stqd_model.py`: discards absolute amplitude cues critical for `dom` |
| **No gradient flow analysis** | Low | No logging of attention weights, gradient norms, or feature importance |

---

## 6. Root Cause Hypotheses (For Investigation)

| # | Hypothesis | Testable Prediction |
|---|------------|---------------------|
| **H1** | **Capacity mismatch**: STQD (~1.2M params) < VLAAI classifier (~3M+). Cross-attn bottleneck. | Increasing width/heads/depth improves synthetic LODO. |
| **H2** | **Pretraining objective misalignment**: Single-speaker match/mismatch ≠ two-talker attention with switches. | Pretraining on two-talker data (Fuglsang, Broderick) or synthetic-switch data improves fine-tune. |
| **H3** | **Temporal bias harms real switches**: +0.20s prior mismatches true EEG-response latency distribution (variable across datasets/subjects). | Removing bias (`stqd_no_time_bias`) helps real switches; learnable bias helps more. |
| **H4** | **Fixed readout token suboptimal**: Token 72 (2.25s pre-end) may not align with decision boundary. | Learnable readout position or multi-token pooling improves both synthetic and real. |
| **H5** | **Per-window standardization destroys `dom` signal**: `dom = (e1-e2)/(e1+e2+ε)` relies on relative amplitude; z-scoring per window per candidate removes this. | Removing per-window standardization for `dom` feature improves performance. |
| **H6** | **Single-scale EEG loses high-freq spatial detail**: Stride-4 pooling discards <8Hz spatial patterns. | Multi-scale EEG encoder (stride 1, 2, 4 branches) improves classification. |
| **H7** | **Cosine > Cross-Attn suggests retrieval paradigm**: `stqd_cosine` (0.59 H) > `stqd_full` (0.54 H) implies matching > interaction. | Pure embedding-based retrieval (Siamese, prototypical) may outperform cross-attn. |

---

## 7. Reviewer Assessment (A/A* Conference Standard)

| Criterion | Score (1-5) | Justification |
|-----------|-------------|---------------|
| **Novelty** | 3/5 | Cross-attn query decoder is clean but components are known (separate encoders, cross-attn, antisymmetry). No new principle. |
| **Empirical Rigor** | 2/5 | Run incomplete (12.5% real-switch coverage); no statistical testing; fusion comparison unfair (2 vs 4 classifiers). |
| **Baseline Comparison** | 2/5 | Classifier-only gap (−0.21 H) not explained; fusion masks it with unequal classifier count. |
| **Real-World Validity** | 3/5 | Compliance checks pass; but temporal bias & fixed readout are unvalidated assumptions for real deployment. |
| **Reproducibility** | 4/5 | Code complete, resumable, seed-controlled, single log; but depends on external data/decoder folds not in repo. |
| **Clarity** | 4/5 | README thorough; code modular; variants well-defined; log complete. |

**Overall Verdict: REJECT (in current state)** — The classifier is not competitive. Fusion results are confounded. Requires either (a) fixing classifier to match baseline on synthetic LODO, or (b) a compelling reason why weaker classifier + strong decoder is a valid paradigm.

---

## 8. Comparison to `decoder_lodo` Baseline

| Metric | `decoder_lodo` (4 dec + 4 clf) | `huyle` (4 dec + 2 clf) | Delta |
|--------|-------------------------------|------------------------|-------|
| **Synthetic LODO (classifier only)** | ~0.75 H | 0.54 H | **−0.21 H** |
| **Synthetic LODO (fusion)** | ~0.80 H | ~0.78 H | −0.02 H |
| **AASD real-switch (fusion)** | ~0.64 H | **0.65 H** | **+0.01 H** |
| **Carta real-switch (fusion)** | ~0.64 H | **0.64 H** | **−0.003 H** |
| **Classifier seeds per variant** | 4 | 2 | −2 |
| **Real-switch eval completeness** | 4 seeds × 3 configs | 2 seeds × 1 config | Incomplete |

> **Interpretation:** The huyle classifier is **not a drop-in replacement**. It only "works" in fusion because the decoder dominates. On Carta (more realistic: visual cues, babble, longer trials), it loses.

---

## 9. Files & Artifacts Inventory

```
/root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/experiments/huyle/
├── Code
│   ├── stqd_model.py              # STQD v1 architecture (183 lines)
│   ├── direct_classifier.py       # Training/eval loop (284 lines)
│   ├── run_all.py                 # Orchestration script (1656 lines)
│   ├── eval_external_clf.py       # Real-switch evaluation
│   ├── fuse_external.py           # Decoder+classifier fusion
│   ├── common_exp.py, config.yaml, decoders.py, score.py, ...  # Copies from decoder_lodo
│   └── tests/                     # Unit tests (shape, antisymmetry, gradients)
├── Logs
│   └── logs/huyle_run.log         # Complete run log (4094 lines)
├── Models
│   └── models/pretrain_spk_direct_clf_huyle_<variant>_s<seed>.pt  # 16 checkpoints
├── Folds (LODO evidence)
│   └── folds/direct_clf__eaP_n5_t60_huyle_<variant>_s<seed>/<DS>.pkl  # 16×5 = 80 files
└── Results
    ├── results/direct_clf__eaP_n5_t60_huyle_<variant>_s<seed>.json  # 16 files
    ├── results/huyle_summary.csv           # Classifier-only LODO summary
    └── results/research/
        ├── ext_clf_AASD_huyle_stqd_full_evidence_s<seed>.pkl
        ├── ext_clf_Carta2026_huyle_stqd_full_evidence_s<seed>.pkl
        ├── ext_clf_AASD_huyle_stqd_full_s<seed>.json
        ├── ext_clf_Carta2026_huyle_stqd_full_s<seed>.json
        ├── fuse_external.csv               # Fusion results (real switches)
        └── fuse_general.csv                # Fusion results (synthetic LODO)
```

---

## 10. How to Resume / Complete the Run

```bash
cd /root/disk_3tb/EEG-sAAD-Grand-Challenge-Training-Dataset/experiments/huyle
PY=../../.venv/bin/python

# Option 1: Complete remaining real-switch evals (skips existing)
PYTHONUNBUFFERED=1 "$PY" run_all.py --skip-lodo --skip-fusion

# Option 2: Run specific variant/seed manually
# Example: stqd_joint_stem seed 0 (where it crashed)
$PY eval_external_clf.py \
    --seed 0 --variant stqd_joint_stem --window-s 10 --prep eaP_n5_t60 \
    --dataset AASD,Carta2026 --tag _huyle_stqd_joint_stem \
    --ref direct_clf__eaP_n5_t60_huyle_stqd_joint_stem_s0 \
    --min-epochs 10 --set max_epochs=40 \
    --p-switch 0.5 --aug-set clf_p_switch2=0.5 \
    --init-from models/pretrain_spk_direct_clf_huyle_stqd_joint_stem_s0.pt

# Option 3: Re-run fusion after all real-switch evals complete
$PY fuse_external.py --dec vlaai_ws__eaP_n5_t60_aug_cluster_onset_dom_xFUG_ptSPK:0 vlaai_ws__eaP_n5_t60_aug_cluster_onset_dom_xFUG_ptSPK_s1:1 vlaai_ws__eaP_n5_t60_aug_cluster_onset_dom_xFUG_ptSPK_s2:2 vlaai_ws__eaP_n5_t60_aug_cluster_onset_dom_xFUG_ptSPK_s3:3 --clf direct_clf__eaP_n5_t60_huyle_<variant>_s0:_huyle_<variant>:0 direct_clf__eaP_n5_t60_huyle_<variant>_s1:_huyle_<variant>:1 --dataset AASD,Carta2026 --dec-tag _best --a 0.0 0.3 0.5 0.7 1.0 --pstay 0.95 0.99 0.998 0.9997 --name huyle_<variant>
```

---

## 11. Recommended Next Steps (For Researcher)

### Priority 1: Diagnose Classifier Weakness
- [ ] Run `stqd_cosine` on real switches (AASD/Carta) — it's the strongest variant (0.59 H synthetic)
- [ ] Ablate per-window standardization for `dom` feature (line 165 `stqd_model.py`)
- [ ] Test learnable readout token position vs fixed 72
- [ ] Add multi-scale EEG encoder (parallel strides 1, 2, 4)

### Priority 2: Fair Baseline Comparison
- [ ] Train 4 seeds per variant (match baseline's 4 classifier seeds)
- [ ] Run full fusion: 4 dec + 4 clf (not 4 dec + 2 clf)
- [ ] Statistical testing: paired t-test across seeds, report confidence intervals

### Priority 3: Real-Switch Generalization
- [ ] Investigate why AASD improves with fusion but Carta doesn't (different switch statistics)
- [ ] Test temporal bias removal (`stqd_no_time_bias`) on real switches
- [ ] Explore domain-adaptive temporal bias (per-dataset learnable)

### Priority 4: Architectural Evolution
- [ ] If cross-attn fundamentally limited, try Siamese/prototypical retrieval (`stqd_cosine` direction)
- [ ] Consider joint EEG-speech encoder with late fusion (current separate encoders may lose joint features)
- [ ] Evaluate whether 10s window is optimal for STQD (baseline uses 10s but different architecture)

---

## 12. Independent Re-Evaluation Checklist

*For the next agent evaluating this work:*

- [ ] Verify all 16 classifier LODO results exist in `results/*.json`
- [ ] Confirm `huyle_summary.csv` matches individual JSONs
- [ ] Check `logs/huyle_run.log` for training anomalies (loss divergence, val BCE plateaus)
- [ ] Validate antisymmetry: swap speech columns → logit sign flip (unit test + manual check)
- [ ] Confirm no data leakage: `zero_eeg` and `shift_eeg` controls at chance (~0.5)
- [ ] Re-compute fusion with equal classifier count (4 dec + 4 clf)
- [ ] Run leakage test on full pipeline including preprocessing
- [ ] Assess whether synthetic-switch LODO is a valid proxy for real-switch performance (evidence: it's not)

---

## 13. Final Judgment

> **The huyle STQD v1 framework is a well-executed research prototype that fails to meet its primary objective: replacing the direct classifier without performance loss. The classifier underfits synthetic-switch LODO by a large margin (0.54 vs 0.75 H), and only appears competitive in fusion because the decoder dominates. Real-switch evaluation is incomplete and confounded by unequal classifier counts.**

-