# EEG-sAAD 2027 — End-to-End Inference Reference

**Baseline:** `experiments/decoder_lodo` (documented best: `eaP_n5_t60`, four 10-second direct classifiers + four 5-second reconstruction decoders, temporal evidence fusion and a two-state HMM).  
**Purpose:** A source-grounded, return-to-it-anytime explanation of what each array means, how its shape changes, and how one continuous trial becomes a 128 Hz attention-label sequence.  
**Scope:** **Inference first**, with a short training appendix so you remember which components are (and aren't) jointly trained.  
**Last source reviewed:** uploaded `decoder_lodo.zip`, `README_BEST.md`, `FILE_FORMATS_AND_SCRIPTS.md`, and the team EDA/onboarding reports (October 2026).

> **Quickest way to refresh your memory:** open [Diagram 1 — Entire system](diagrams/01_complete_inference_overview.svg), then [Diagram 4 — Timing alignment](diagrams/04_temporal_alignment_indices.svg), then read §1, §5, and §8 below. For concrete numbers, run `python toy_inference_trace.py`.

## Contents

1. [Five essential definitions](#1-five-essential-definitions)
2. [The complete drawing](#2-the-complete-drawing)
3. [From organizer files to one prepared trial](#3-from-organizer-files-to-one-prepared-trial)
4. [Direct classifier: complete inference path](#4-direct-classifier-complete-inference-path)
5. [Reconstruction decoder: complete inference path](#5-reconstruction-decoder-complete-inference-path)
6. [Alignment: same evidence time, different input duration](#6-alignment-same-evidence-time-different-input-duration)
7. [LDA, ensemble fusion, HMM, and final output](#7-lda-ensemble-fusion-hmm-and-final-output)
8. [Exact-size 20-second walkthrough](#8-exact-size-20-second-walkthrough)
9. [Numerical mini-example: 8 time samples](#9-numerical-mini-example-8-time-samples)
10. [What differs during training](#10-what-differs-during-training)
11. [Causality and important implementation caveats](#11-causality-and-important-implementation-caveats)
12. [Source map and glossary](#12-source-map-and-glossary)

---

## 1. Five essential definitions

- **Trial** = one continuous recording from one listening experiment. It has many EEG time samples, the two candidate speech streams, and—**only for labeled training/evaluation**—ground-truth attention labels. A trial is **not** one network window.
- **Window** = a temporal crop from one trial. The direct classifier sees **10 seconds** (`320` samples at 32 Hz); the reconstruction CNN sees **5 seconds** (`160` samples at 32 Hz).
- **Batch** (`B`) = how many windows are passed through a neural network together. During training these can come from **different trials**. During inference all output sequences are assembled **independently for the same single test trial**.
- **Hop** (`H=8` at 32 Hz) = move forward **0.25 seconds** between successive inference windows/evidence calculations. **A 10-second input window does not mean a prediction once every 10 seconds.**
- **Label time versus availability time:** an attention decision is associated with an earlier time than the latest EEG sample needed to calculate it. The task allows at most **1 second of future EEG** for a label at time `t`. The 128 Hz final output resolution is **not** the evidence-update frequency.

### The three different temporal resolutions

| Quantity | Production value | Interpretation |
|---|---|---|
| Original data/output | **128 Hz** | Required binary attention label for every 7.8125 ms |
| Prepared input | **32 Hz** | One EEG/acoustic-feature value every 31.25 ms |
| Decision/evidence hop | **0.25 s = 8 samples at 32 Hz** | The evidence/HMM updates **four times per second** |
| Decoder correlation window | **1 s = 32 samples** | Each **one** reconstruction evidence value compares **32 reconstructed samples** with both speakers |

The official task **does not** require one prediction per second. One second is a **correlation window**, not the output granularity.

## 2. The complete drawing

**Overview — from CND input to the final 128 Hz array:**

![Entire inference framework](diagrams/01_complete_inference_overview.svg)

**Direct classifier — down to individual convolutions and the two speaker orderings:**

![Direct classifier detail](diagrams/02_direct_classifier_details.svg)

**Reconstruction decoder — how it reconstructs, crops, stitches, compares and yields evidence:**

![Reconstruction detail](diagrams/03_reconstruction_decoder_details.svg)

**Temporal alignment — sample indices, the +3 shift and the distinction between availability and label time:**

![Timing detail](diagrams/04_temporal_alignment_indices.svg)

All figures are also available as high-resolution PNGs, and their `.dot` source files are editable using Graphviz (`dot -Tsvg file.dot -o file.svg`).

## 3. From organizer files to one prepared trial

### 3.1 Actual data provenance

The organizer's **extended CND** recording uses separate MATLAB subject/stimulus structures. A given trial's original EEG is `[T128, C_raw]`, its two original speech envelopes form `[T128,2]`, and known training labels are `[T128]` (stream or azimuth depending on target). `C_raw` is dataset-specific: four datasets have 64-channel recordings; KUL-UHD is ultra-high density.

There are **five labeled training datasets**: KUL-AV-GC, DTU, KUL2016, KUL-UHD and USTC. The team reports 95 subjects and 1,979 trials, approximately 77 hours. Their **attended speaker identity stays fixed within each official training trial**. KUL-AV-GC changes the attended talker's *spatial side* at 300 s, **not the attended voice identity**. That distinction matters when interpreting synthetic speaker-switch augmentation and validation scores.

The documented best preprocessing path is `eaP_n5_t60`:

1. Load the organizer's `pre_*.mat` EEG/envelopes: supplied zero-phase 1–9 Hz bandpass, bad-channel/segment repair, **32 Hz**. (The organizer's preprocessing is a documented exception to the general future-context rule; do **not** assume arbitrary zero-phase operations are permitted.)
2. Align named electrodes to the **58-channel common montage**; fill missing channels consistently.
3. Apply **common-average reference (CAR)** across present electrodes.
4. Trim **1 second from each end** of the prepared trial. The prepared cache retains `t0`, an offset into the original trial.
5. Standardize EEG by channel and the two envelopes using one **whole-trial z-score** operation under the team's documented exemption.
6. Apply **causal running Euclidean alignment** to EEG, updating spatial covariance from past one-second blocks, initialized from a fixed population covariance prior; warm-up prior weight `5 s`, time constant `60 s`, shrinkage `0.1`. This changes values but not the `[T,58]` shape.
7. Construct speech features: **envelope**, positive **onset** (first difference), **relative dominance** (`dom`, from the team's separate causal feature builder).

Final prepared **one-trial data structure**:

```text
trial['eeg'] = X                 float32 [T,58]     # prepared 32 Hz
trial['env'] = E                 float32 [T,2]      # 2 speaker envelopes
_targets(trial) = S              float32 [T,2,3]    # env,onset,dom per stream
trial['att'] = y                 int8    [T]        # TRAIN/evaluation only; not test input
trial['t0'] = t0                 scalar             # original-trial offset
```

**Important column convention:** SuperHuge training conversions often place the originally attended stream in column `0`. This is a property of training conversion and must **not** be treated as a predictive rule on unseen trials. `att_col` restores source mappings for evaluation and fitting.

### 3.2 Why `S:[T,2,3]` and not `[2,3]`?

Every speaker has **three features at every 32 Hz sample**. `[T,2,3]` preserves acoustic fluctuations and their alignment with the EEG. `[2,3]` would collapse all temporal variation and discard the neural–speech timing relationships. In the direct model's one 10-second window, the full speech input is `[320,2,3]`.

---

## 4. Direct classifier: complete inference path

**One trial, one trained model/seed.** The input to the neural network **is not the entire `[T,58]` trial at once**: instead we collect overlapping **10-second EEG/speech windows** on the 0.25-second grid. Processing may be minibatched to reduce memory usage.

Constants at 32 Hz:

```python
Wc  = 320                  # 10 s
H   = 8                    # 0.25 s
F   = 16                   # 0.50 s (future parameter)
Pc  = Wc - F - H  # 296     # left padding / readout offset
LAB = Wc - 32     # 288     # TRAINING supervision location within each window
```

For a hop indexed by `i`, with `a=8*i`:

```text
EEG slice:      X[a - 296 : a + 24]  → [320,58], zero-padded as needed
Speech slice:   S[a - 296 : a + 24]  → [320,2,3], same padding
Neural label location: local index 288 → original prepared index (a - 8)
Latest EEG exclusive boundary: (a + 24)
```

Per **batch of B windows**:

| Tiny operation (`direct_classifier.py` / `vlaai_ws_model.py`) | Shape |
|---|---|
| EEG inputs | `[B,320,58]` |
| Both speakers' speech features | `[B,320,2,3]` |
| Standardize each speech feature over its 320 samples, per speaker/example | `[B,320,2,3]` |
| Concatenate `[EEG, S_A, S_B]` | `[B,320,64]` |
| Switch to Conv1d `[B,channels,time]` | `[B,64,320]` |
| Input projection, Conv1d kernel 1 | `[B,64,320]` |
| Shared-block pass 1: temporal kernels **3,7,11**, context kernel **17**, ChannelMix | `[B,64,320]` |
| Shared-block pass 2, input skip, **same weights** | `[B,64,320]` |
| Shared-block pass 3, input skip, **same weights** | `[B,64,320]` |
| Transpose | `[B,320,64]` |
| Linear per-position readout, 64 → 1 | `[B,320,1]` |
| Remove singleton and learned softmax-weighted pool across **all 320 positions** | `[B]` |
| Evaluate network again with speaker order reversed | another `[B]` |
| **Antisymmetric logit** `g(X,A,B)-g(X,B,A)` | **`[B]`** |

**Why the order swap?** Mathematically it enforces `logit(X,B,A) = −logit(X,A,B)`. It prevents a constant fixed-column bias from directly explaining labels; it **doesn't alone prevent every acoustic-only shortcut**.

### What does one direct logit mean?

It is supervised to identify the **attended candidate at a designated position** of the window, not a majority vote over ten seconds and not a forecast of the *next* 0.25 s. During training the label is taken at local position `LAB=288` (9 s from window start). During inference hop `a`, that maps to `a−8`, i.e. **0.25 s earlier than the hop's `a` timestamp**. More recent EEG up to `a+24` is permitted future context.

If the prepared trial contains `T=640` samples (20 s), and `T` is divisible by 8, there are **80 classifier hops**, each with one logit: `C:[80]` **per seed**. The best ensemble contains **four separately trained seeds**, so conceptually `[4,80]` before fusion. These are **80 successive decisions for one trial**, not one decision for the full trial.

---

## 5. Reconstruction decoder: complete inference path

This is a different kind of model. Its CNN **only sees EEG**; the candidate speech features are needed **after reconstruction** for inference-time comparison. During training they supervise the reconstruction loss.

Constants:

```python
Wr=160                     # 5 s EEG input
H=8                        # 0.25 s output block / inference hop
F=16                       # 0.50 s FUTURE after selected output block
Pr=Wr-F-H       # 136      # selected readout starts here, within the model window
```

For a reconstruction block beginning at prepared-trial sample `a=8*k`:

```text
Input EEG: X[a−136 : a+24]  [160,58] (boundary padding as necessary)
VLAAI backbone             [B,160,58] -> [B,160,64]
4 passes shared blocks     [B,160,64] -> [B,160,64]
Linear head 64 -> 3        [B,160,3]   # all 160 temporal reconstructions
TAKE ONLY                  [:,136:144,:] = [B,8,3]
Place into trial output    Yhat[a:a+8,:] = output[:,136:144,:]
```

**Why only eight reconstructed samples?** Because the hop is **eight samples**: we want to contribute one new contiguous **0.25-second reconstructed segment** at each step. If we copied all 160 positions from every overlapping window, most trial timestamps would receive many conflicting estimates from differing context boundaries.

**Why positions 136–143 rather than the last eight (152–159)?** Their input window consists of:

```text
   136 preceding rows     8 predicted rows       16 later EEG rows
   ├───────────────────────┼──────────────────────┼───────────┤
     historical context      KEEP [136:144]         future
```

`F=16` means the 16 EEG samples after that selected block are available as neural-response-lag context. For the earliest retained output, **up to 23 later samples** (0.71875 s) are present; for the last retained output, 16 later samples (0.5 s). In all cases, the latest allowed EEG boundary is `< a+24`, within the task's one-second limit. The network's internal convolutions are not individually causal; this **window/readout schedule** imposes the input-information boundary.

**Stitching:** at `a=0` retain output `[0:8]`, at `a=8` retain `[8:16]`, etc. For `T=640`, 80 blocks × 8 rows = **`Yhat:[640,3]`**. This is still reconstructed speech features, **not** speaker decisions.

### Converting `[640,3]` into `[77]`: why exactly 77?

The evidence estimator compares the **reconstructed features** `Yhat` against the **actual features of both available candidate speakers** on **1-second (=32-sample) windows**, moved every **8 samples**:

```text
window j=0:   [0:32]    reconstructed vs A, reconstructed vs B
window j=1:   [8:40]
window j=2:   [16:48]
...
window j=76:  [608:640]
```

Number of **complete** evidence windows:

\[
N_E = \left\lfloor\frac{T-32}{8}\right\rfloor+1
\]

For `T=640`, `N_E=(640−32)/8+1=**77**`.

For each window, compute six Pearson correlations: **3 features × 2 speakers**. Subtract candidate correlations:

\[
\Delta r_{j,f} = \mathrm{corr}(\hat Y_{j,f}, S_{A,j,f})-\mathrm{corr}(\hat Y_{j,f}, S_{B,j,f})
\]

`ΔR:[77,3]` (for env/onset/dom). An **LDA** weight vector `w:[3]`, fitted on signed training-domain evidence, maps `ΔR @ w → R:[77]` **per decoder seed**. The scalar's **sign** favors speaker column 0 or 1; it is **not a calibrated probability**.

The LDA weights are obtained approximately using `w ∝ Σ⁻¹ μ` from training folds (`score.lda_weights`, `fuse_general.build`). A weighted correlation difference isn't directly a binary classification logit.

---

## 6. Alignment: same evidence time, different input duration

**Central lesson:** Never try to concatenate or align the classifier's `[320,58]` *input tensor* with the reconstruction network's `[160,58]` *input tensor*. We only align their **scalar evidence sequences**, computed from **the same trial**.

Classifier evidence at hop `i` consumes EEG:

\[
X[8i-296\;:\;8i+24],\qquad \text{latest boundary}=8i+24
\]

Reconstruction evidence window `j` uses reconstructed samples `[8j:8j+32]`; its last reconstructed block needs EEG up to exclusive boundary `8j+48`:

\[
\text{latest boundary for reconstruction evidence }j = 8j+48
\]

Match these boundaries:

\[
8i+24=8j+48\quad\Longrightarrow\quad \boxed{i=j+3}
\]

Thus source `fuse_evidence.py` defines **`DEC_OFFSET=3`**:

```text
reconstruction R[0]  <-> classifier C[3]
reconstruction R[1]  <-> classifier C[4]
...
reconstruction R[76] <-> classifier C[79]

raw C:[80] -> use C[3:80]:[77]
raw R:[77] -> use R[0:77]:[77]
```

For the first aligned pair `j=0`:

- Reconstruction correlations use reconstructed samples `[0:32]` (first second of the trial).
- The decoder needs EEG **up to sample 48, exclusive**.
- Classifier hop `i=3` extracts EEG `[-272:48]` (zero-padded before start); it also needs EEG **up to sample 48, exclusive**.
- The corresponding **attention-label time** is `(8j+16)/32 = 0.50 s` after the prepared trial's start.
- The **availability boundary** is `(8j+48)/32 = 1.50 s`: the same label uses at most **1 second** future EEG. The actual last observed sample is 47, so strict sampled lookahead is 31/32 s.

**Do not confuse the one-second future-context budget with the one-second correlation window or the 0.25-second evidence hop.** These are three independent design quantities.

This alignment is based on the **latest allowable input sample**, not identical historical spans. The direct branch uses 10 s of historical context; the reconstruction branch uses 5 s and then a 1 s correlation of reconstructed features. They can still speak about the **same hidden attention state at the same effective label time**.

### Output timestamp placement

The actual `score.py` path calls `e1_switch.expand(..., t_end, lookahead, original_length, initial_guess)` and documents a stamp at:

\[
\text{label time}=t_{\mathrm{end}}+\mathrm{lookahead}-1\text{ s}
\]

For **fused** evidence the `t_end` is taken from **the aligned classifier** hop `i=j+3`, namely `(8j+24)/32 + t0`. The fused `lookahead` parameter is `(16+8)/32=0.75 s`. Hence:

\[
\boxed{t_{\mathrm{label},j}=t0+(8j+16)/32}\quad\text{(fused path)}
\]

That means j=0 stamps **t0+0.50s**, j=1 **t0+0.75s**, etc. A `t0` offset can be present because the organizer's prepared files trim trial edges.

**Caveat:** `e1_switch.py` and the production blind-submission exporter were **not** in the uploaded `decoder_lodo.zip`. The stamping **formula and call signature** are documented/visible in `score.py`, while exact `expand()` boundary behavior (random initial guess, tail mapping, rounding/edge cases) cannot be verified line-by-line from this archive. The toy script uses an explicitly stated initial-guess policy rather than pretending to reproduce that missing implementation.

---

## 7. LDA, ensemble fusion, HMM, and final output

### 7.1 What is being fused?

Four direct-classifier seeds produce `C^(s):[N_C]`, while four reconstruction-decoder seeds produce `R^(s):[N_E]`, **after LDA**. On each held-out LODO fold, the downstream scales and weights are based on evidence from **the four other datasets' out-of-fold results**, not the held-out labels.

The documented ensemble (`fuse_general.build`) performs:

1. Divide each seed's scalar evidence by its training-domain standard deviation.
2. Average four normalized seed sequences within each branch.
3. Divide each group average by its group-level training-domain standard deviation.
4. Align classifier `[N_C]` and reconstruction `[N_E]` sequences with the three-hop offset.
5. Weighted average by `alpha`, chosen as `0.5` in one reported best configuration:

\[
 d_j = (1-\alpha)z_{R,j} + \alpha z_{C,j}
\]

**`d:[N_E]` is a sequence of signed fused attention-evidence scores**, not one scalar for the trial. Its sign favors a candidate speaker, but its magnitude is **not an automatically calibrated probability**, and the reconstruction component wasn't necessarily a logit.

### 7.2 What is HMM doing?

The **Hidden Markov Model** treats actual attention as a hidden state `s_j∈{A,B}` and the fused score `d_j` as a noisy observation. It favors continuity of attention but can switch when accumulated evidence is convincing.

- Transition probability: `P(stay)=0.998` **per 0.25-second evidence step** (best documented setting).
- Gaussian emissions: `d|A ~ N(+μ,σ²)` and `d|B ~ N(−μ,σ²)`, with μ and σ fitted from training-domain signed evidence.
- Forward filtering updates `posterior(s_j)` from `posterior(s_(j−1))` and the new observation `d_j`. It does **not** look ahead through future evidence.
- Predicted state is the higher posterior at each available evidence step.

\[
\tilde p_j(s)=\sum_{s'}p_{j-1}(s')P(s\mid s'),\quad
p_j(s)\propto \tilde p_j(s)\,\mathcal N(d_j;\mu_s,\sigma^2)
\]

The HMM is a mathematically sound **approximate temporal integrator**. Its Markov and Gaussian assumptions are imperfect, especially because 1-second evidence windows overlap by 75%, while adjacent 10-second classifier windows overlap even more. Thus HMM posteriors are not guaranteed calibrated.

### 7.3 What finally gets submitted?

The model predicts an HMM state at each quarter-second evidence step. `expand()` assigns state predictions to the appropriate **128 Hz sample intervals**, with initial/boundary handling and original-trial offsets. A 20-second output span requires **2,560 labels**, even though the fused evidence in our 20-second prepared example contains only 77 scores.

```text
Fused evidence [77] -> HMM states [77]
                  -> timestamped piecewise-constant attention curve
                  -> 128 Hz array [T128] with values 0 or 1
```

**Attention classification, not audio reconstruction, is the submission target.** The reconstructed envelope/onset/dom are intermediate representations.

---

## 8. Exact-size 20-second walkthrough

Here `T=640` means **20 seconds of already prepared 32 Hz data**. Ignore the original-trial `t0` for counting; any real crop offset must be restored at export.

| Step | Object | Shape | Why |
|---:|---|---|---|
| 0 | Prepared EEG trial | `[640,58]` | 20 s × 32 Hz, 58 common electrodes |
| 1 | Prepared speaker envelopes | `[640,2]` | two candidate streams |
| 2 | `S = _targets(trial)` | `[640,2,3]` | env/onset/dom for each speaker |
| 3C | Direct EEG window collection | `[80,320,58]` | 80 hop starts, 10 s per window, zero padded |
| 4C | Direct speaker window collection | `[80,320,2,3]` | same hop starts, same 10 s spans |
| 5C | Direct combined channels | `[80,320,64]` | 58 EEG + 6 speech features |
| 6C | Direct `g()` outputs, both speaker orderings | `[80]`, `[80]` | one pooled scalar per input window per ordering |
| 7C | Subtracted direct logits | `[80]` | `C_i = g(X,A,B)−g(X,B,A)` |
| 3R | Decoder EEG window collection | `[80,160,58]` | 80 hop starts, 5 s per window |
| 4R | Decoder full reconstructed windows | `[80,160,3]` | three acoustic targets at each of 160 positions |
| 5R | Retained reconstruction blocks | `[80,8,3]` | only positions 136–143 of each window |
| 6R | Stitched reconstruction `Yhat` | `[640,3]` | 80×8 positions, one reconstructed sequence |
| 7R | Candidate speech target arrays | `[640,3]` and `[640,3]` | A and B, **not** direct CNN inputs |
| 8R | One-second Pearson correlations | `[77,3]` each candidate | `(640−32)/8 + 1 = 77` |
| 9R | Feature-difference vectors | `[77,3]` | `corr(Yhat,A)−corr(Yhat,B)` |
| 10R | LDA-weighted reconstruction evidence | `[77]` | dot product with trained `[3]` weights |
| 11 | Align direct logits | `C[3:80] -> [77]` | `i=j+3` same latest EEG sample |
| 12 | Fuse branch evidence | `[77]` | after training-derived scale normalization |
| 13 | Two-state HMM posterior | `[77,2]` | probabilities for A and B over evidence steps |
| 14 | HMM selected state | `[77]` | discrete state per step |
| 15 | Stamp, expand, restore original timeline | `[2560]` **if** full output span = 20 s | 128 Hz, piecewise-constant sequence |

The window collections above are **conceptual full collections**. Actual code batches window processing (e.g. `direct_classifier.logits` processes chunks up to 1024; reconstruction `DNNDec.reconstruct` up to 512). If `T` is not divisible by `H`, the last reconstructed block is cropped and `N_C=ceil(T/H)`; evidence windows remain `floor((T−32)/8)+1` when `T≥32`.

### First two temporal correspondences for the 20-second trial

| Reconstruction evidence | Correlation block | Latest EEG exclusive boundary | Classifier index | Classifier EEG window | Label timestamp |
|---|---|---:|---:|---|---:|
| `R_0` | reconstruction `[0:32]` | 48 | `C_3` | `X[-272:48]` | `t0+0.50 s` |
| `R_1` | reconstruction `[8:40]` | 56 | `C_4` | `X[-264:56]` | `t0+0.75 s` |
| `R_2` | reconstruction `[16:48]` | 64 | `C_5` | `X[-256:64]` | `t0+1.00 s` |

Negative indices are **padded zero history**, not actual samples or native negative-index slicing. Each pair's latest boundary is **32 prepared samples (1 second)** after its label index.

---

## 9. Numerical mini-example: 8 time samples

The runnable [toy_inference_trace.py](toy_inference_trace.py) calculates the numbers below; see its [captured output](toy_inference_trace_output.txt). **All neural-network outputs and model parameters are explicitly synthetic**. The purpose is to trace shapes, correlation arithmetic, indexing, HMM updates and final expansion, not to approximate true EEG physiology. In particular, the toy rate of 4 Hz cannot represent the actual 1–9 Hz EEG feature band.

### 9.1 Toy configuration and inputs

- Toy EEG/sample rate: **4 Hz**; output still **128 Hz**.
- EEG: two electrodes, `X:[8,2]`, a **2-second trial**.
- Two candidate envelopes: `E:[8,2]`. Derived demonstration features create `S:[8,2,3]`.
- Toy direct window `Wc=8`; decoder window `Wr=4`; correlation window `We=4`; hop `H=1` (0.25 s); future `F=2` (0.5 s).
- **Ground truth for evaluation only:** first second attends candidate A (`0`), second second candidate B (`1`). A real inference process **does not receive** these labels.

```text
EEG X [8,2]                    Candidate speech envelopes [8,2]
index   EEG1  EEG2             index     A    B
0        .8    -.2              0        0    2
1        .6    -.1              1        1    1
2        .9     0               2        2    0
3        .7    -.2              3        1    1
4       -.3     .8              4        0    2
5       -.2     .9              5        1    1
6       -.1     .7              6        2    0
7       -.4     .6              7        1    1
```

### 9.2 Synthetic direct classifier produces eight logits

With `Pc=Wc−F−H=5`, its first window is `X[-5:3]` (5 padded rows + real rows 0,1,2). Sliding by `H=1`, we get `8` input windows of shape `[8,2]` and **one scalar logit each**:

```text
C:[8] = [+0.30, +0.35, +0.25, +0.90, +0.80, +0.10, −0.40, −0.70]
```

These eight scalar values are **illustrative model outputs**, not computed from toy EEG by an actual trained CNN.

### 9.3 Synthetic reconstructed feature sequence

With `Pr=Wr−F−H=1`, each decoder window of shape `[4,2]` produces `[4,3]` and contributes only **local row 1**, one of the 8 trial rows, to `Yhat:[8,3]`. The toy reconstructed **envelope** is:

```text
Yhat[:,0] = [0,1,2,1, 2,1,0,1]
```

For this toy only, the code computes onset and a simple complementary-stream dominance analogue, creating a concrete `[8,3]` reconstruction. The real `dom` feature in `build_glimpse.py` uses a different causal processing chain.

### 9.4 Five one-second correlation windows

A window contains `We=4` time samples, stride `H=1`:

```text
j0: [0:4]  j1: [1:5]  j2: [2:6]  j3: [3:7]  j4: [4:8]
number = (8−4)/1+1 = 5
```

Compute Pearson correlations between `Yhat` and both candidate-feature sequences, then subtract. The **actual computed** three-feature differences (rounded) are:

\[
\Delta R=\begin{bmatrix}
+2.000&+1.577&+2.000\\
0&+1.155&0\\
0&0&0\\
-2.000&-1.155&-2.000\\
-2.000&-2.000&-2.000
\end{bmatrix}\quad[5,3]
\]

Use **illustrative** fixed LDA weights `w=[0.3,0.4,0.3]`:

```text
R = ΔR @ w  =  [1.831, 0.462, 0.000, −1.662, −2.000]  shape [5]
```

A correlation **difference** is in `[-2,+2]`, because each individual Pearson coefficient is in `[-1,+1]`.

### 9.5 Align `C[8]` with `R[5]`, then fuse

Same offset rule: `i=j+3`. Select `C[3:8]`:

```text
C_aligned:[5] = [+0.90,+0.80,+0.10,−0.40,−0.70]
R:[5]         = [+1.831,+0.462,0.000,−1.662,−2.000]
```

As a toy calibration, divide `R` by 2 and weight the two branches equally:

\[
d_j=0.5(R_j/2)+0.5\,C_{j+3}
\]

```text
Fused d:[5] = [+0.908,+0.515,+0.050,−0.615,−0.850]
```

Those are **five scalar evidence scores over time**, **not one logit for the entire trial**.

### 9.6 HMM, timestamps and 128 Hz expansion

Toy two-state HMM: symmetric Gaussian observation means `±0.4`, standard deviation `0.6`, `p_stay=0.8` per step (deliberately *not* the production `0.998`, so a two-second example can switch). Starting posterior `P(A)=0.5`:

```text
step             0      1      2      3      4
P(A)           .883   .895   .758   .326   .090
HMM states        A      A      A      B      B
label times(s)   .50    .75   1.00   1.25   1.50
```

The first prediction is stamped to 0.50 s, although its calculation requires EEG through the boundary at 1.50 s. In this toy example, we **assume the initial guess is A** for `[0,0.50)` and hold the last state to trial end. The exact official `expand()` implementation is not part of the supplied archive.

A two-second trial needs `2×128=256` labels. Each 0.25 s segment is 32 labels:

```text
Ground truth quarter-second blocks:  A A A A B B B B
Predicted quarter-second blocks:     A A A A A B B B
Final labels [256]:                  0×160, then 1×96
Sample accuracy (this toy only):      224/256 = 87.5%
```

**Everything above can be verified with `python toy_inference_trace.py`.** The script asserts the shapes, offset, timestamps, HMM decisions, output length and accuracy. Its stored stdout is `toy_inference_trace_output.txt`.

---

## 10. What differs during training

Inference must align the two branches because both produce evidence about the **same test trial**. **Training does not require their sampled minibatches to share a trial or a timestamp.**

| | Direct classifier training | Reconstruction decoder training |
|---|---|---|
| Sampling | Random eligible training trial + random **complete** 10 s crop | Independent random eligible training trial + random **complete** 5 s crop |
| EEG minibatch | `[B,320,58]` | `[B,160,58]` |
| Speech information | `[B,320,2,3]` **enters the CNN** alongside EEG | `Yatt,Yign:[B,160,3]` **do not enter the CNN**; they supervise its output |
| Network output | `[B]` antisymmetric candidate comparison | `[B,160,3]` reconstructed attended speech features |
| Loss | `BCEWithLogits` at label position `LAB=288` (a single label **per window**) | Mean contrastive correlation `−corr(Yhat,Yatt)+corr(Yhat,Yign)` over all 160 positions |
| Augmentations | EEG augmentation, role-swap within windows, optional second role-swap | EEG augmentations; no ordinary per-window classifier-style `sw2` step |
| Gradient | BCE → direct network only | Pearson loss → reconstruction network only |

**After** fitting individual networks, the pipeline estimates LDA weights, evidence standard deviations, HMM Gaussian parameters and evaluates fusion on out-of-fold, *labeled training-domain evidence*. These are fitted downstream statistical parameters and selected hyperparameters, **not a jointly differentiable loss that backpropagates through both branches**.

The best documented result is roughly `H≈0.80` on **synthetic-switch** LODO evaluation (e.g. `H=0.7981` for 50/50 fusion, `p_stay=0.998`; another weighting/persistence choice yielded `0.8003`). These **are not 80% accuracy and are not blind-test results**. Real-switch external evaluations were appreciably worse (around `H≈0.64` in the team's report), so this should be treated as a **baseline for further research**.

---

## 11. Causality and important implementation caveats

1. **Input window ≠ prediction interval.** The direct 10 s window's single logit predicts the attended candidate **at its designated label index**, not the majority attended voice over the window. Inference repeats this every 0.25 s. The reconstruction CNN outputs an **acoustic time series**, and only the *later* correlation/LDA step turns it into attention evidence.
2. **Future context is intentional.** The rule permits at most 1 s future information. The relevant future budget counts **the entire preprocessing/network/evidence/HMM chain**. The best pipeline's alignment was engineered so fused label `j` needs EEG no later than label time + 1 s (exclusive boundary).
3. **The networks are not inherently causal convolutions.** Their temporal kernels may see later positions within a supplied window; selecting the input span and kept output region is how the *external* causal budget is enforced. The team has a separate `leakage_test.py` for perturbation checks.
4. **The best classifier is a 10 s override.** `config.yaml` defaults `dnn.window_s: 5.0`; `direct_classifier.py --window-s 10` produces the selected best 10 s model. Don't assume every checkpoint has the same window length.
5. **Kernel configuration caveat.** The instantiated source's standard three extractor convolutions have kernels **3,7,11**, followed by an **output-context** convolution with kernel 17. A fourth listed kernel size 15 is not instantiated when the configured extractor layer count is three.
6. **Pearson windows are complete at inference.** For prepared `T<32`, there are no complete one-second windows in the ordinary reconstruction-evidence function. A production system must define the boundary case explicitly.
7. **Initial and final trial segments need handling.** The first complete fused evidence is stamped around `t0+0.5 s` and the last complete one around `t0+T/32−0.5 s`, for a `T` divisible by 8. The scorer's `expand()` handles extra intervals; its exact implementation is **not in this uploaded zip**.
8. **The preparation trim matters.** A prepared `[T,58]` cache usually omits 1 s at each end of the original trial. `t0` and the original 128 Hz length are necessary to map predictions onto the **full** submission timeline. A prepared 20 s trial does not automatically correspond to an original 20 s trial.
9. **Speaker identity versus direction.** In KUL-AV-GC, the attended **side** changes at 300 s but the attended **voice** doesn't. Do not silently mix attended-stream labels, original CND stream numbering, SuperHuge candidate columns and attended azimuth labels.
10. **Avoid accidental leakage in fusion fitting.** The proper LODO design uses evidence from the four *non-held-out* datasets to learn downstream weights/scales/emissions. Labels for the blind dataset must never enter calibration.
11. **The HMM is approximate.** Gaussian emissions + high persistence are reasonable but not perfectly calibrated, especially since overlapping evidence windows are correlated. Hyperparameters tuned on synthetic switches need real-switch testing.
12. **The standalone final submission runner wasn't included.** This package documents the *development inference/scoring path*; it should not be mistaken for proof that the team's planned external blind-test export has been fully tested.

---

## 12. Source map and glossary

### Where each behavior is implemented

| Concept | Uploaded code source |
|---|---|
| Cache/preprocessing/causal EA | `prepare_data.py` (`causal_ea`, `zscore_trial`, cache writers) |
| Per-trial loading | `common_exp.py` (`load_trial`) |
| Acoustic target tensor | `decoders.py` (`_targets`, `_att_tgt`); `build_glimpse.py` for relative dominance |
| Direct network, window sampling and inference logits | `direct_classifier.py` (`DirectNet`, `windows`, `logits`) |
| VLAAI backbone internals | `vlaai_ws_model.py` (`VLAAIwsBackbone`, `VLAAIwsDecoder`) |
| Reconstruction training and causal stitched reconstruction | `decoders.py` (`DNNDec._windows`, `fit`, `reconstruct`, `features`) |
| Sliding Pearson evidence and timestamp `t_end` | `common_exp.py` (`sliding_corr`, `evidence_from_features`) |
| LDA target weighting | `score.py` (`lda_weights`), `fuse_general.py` (`build`) |
| Synthetic-switch alignment / `DEC_OFFSET=3` | `fuse_evidence.py` (`features`) |
| Four-seed group standardization + mixing | `fuse_general.py` (`build`, `one`) |
| HMM scoring/expansion calls | `score.py` (`score_dataset`), external `e1_switch.py` **not supplied** |
| Best experiment description and cautions | `README_BEST.md` |

### Glossary

- **`T128`**: original/evaluation time samples at **128 Hz**.
- **`T`**: trial length after 32 Hz preprocessing and trimming (time samples, not seconds).
- **`B`**: neural-network batch size (windows processed together).
- **`C`**: EEG electrode-channel count, **58** after harmonization.
- **`K`**: neural-network hidden feature-map count, **64** in the baseline.
- **`F_features`**: acoustic features per candidate, **3** (`env`, `onset`, `dom`); distinct from the future-context parameter `F=16`.
- **`Wc=320`**: 10 s direct-classifier input context.
- **`Wr=160`**: 5 s reconstruction CNN input context.
- **`We=32`**: 1 s Pearson evidence comparison window.
- **`H=8`**: 0.25 s sliding hop (number of 32 Hz samples).
- **`F=16`**: 0.5 s future-context parameter (number of 32 Hz samples).
- **`Pc=296`, `Pr=136`**: input/readout offsets, `window_length−F−H`.
- **`N_C` / `N_R`**: number of classifier/reconstruction inference hops, `ceil(T/8)`.
- **`N_E`**: number of **complete** 1 s correlation evidence windows, `floor((T−32)/8)+1` if `T≥32`.
- **`Yhat`**: EEG-derived **reconstructed acoustic features**, not hard attended-speaker labels.
- **`Yatt` / `Yign`**: real attended/ignored candidate acoustic features selected using **ground-truth labels during training**, never known at blind inference.
- **`R_j`**: reconstruction evidence scalar at step `j` after correlation differences and LDA.
- **`C_i`**: direct-classifier antisymmetric scalar logit at hop `i`.
- **`DEC_OFFSET=3`**: pairing `R_j ↔ C_(j+3)`.
- **`d_j`**: fused signed evidence score at aligned step `j`.
- **HMM**: temporal probabilistic integrator producing posterior A/B state estimates from `d_0…d_j`.
- **`t0`**: time offset to recover locations within the untrimmed original trial.

### Future work (not implemented by this reference)

This is a **documentation/teaching package, not a new model release**. Potential research directions (such as calibration-aware fusion, correlated-observation HMM alternatives, differentiable joint training, better real-switch augmentation and test-time adaptation) should be designed and evaluated separately after the baseline behavior above is reproduced and verified.
