# Cross-dataset sAAD scoring

Scores per-sample attention-decoding label files against ground truth, producing four
numbers: **Acc**, **SDT**, **L**, and the headline **H**.

This package is **Part 2 (score)**. Part 1 (`TOPUBLISH-crossDataset-example`) decodes EEG and
writes the CSVs; this half reads those CSVs **and nothing else** — not the EEG, not the
decoder, not any struct the other package produced. So predictions made by any other method,
toolbox, or language are scored on exactly the same footing by pointing it at a directory,
and you can re-score without re-decoding.

## Run it

**`crossDatasetScoring.m` is the main script — the only file you need to call.** It puts the
folder on the path itself.

```matlab
>> crossDatasetScoring
```

Set `outRoot` inside it to the directory of CSVs to score (the `outRoot` Part 1 wrote to, or
any directory laid out the same way), then run.

To score one directory directly:

```matlab
scores = scorePredictions(fullfile(outRoot,'plain'), fullfile(outRoot,'groundtruth'));
```

### Requirements

MATLAB R2019b or newer. **Base MATLAB only** — no toolboxes at all, since this half never
touches a signal.

## Expected input

```
<outRoot>/groundtruth/GTlabels_test_sample_001.csv   the reference
<outRoot>/plain/test_sample_001.csv                  one prediction set
<outRoot>/causal/test_sample_001.csv                 another prediction set
```

One directory is **one test set**, one file is **one subject**. Every file has the same three
**0-based** columns:

```
time_index,label,trial
0,0,0
1,0,0
...
```

* `time_index` — sample index *within its trial*, restarting at 0 for each trial
* `label` — attended/decoded stream (`0` = first stream, `1` = second)
* `trial` — trial index

The default sampling rate is **128 Hz**; the format does not record it, so it is a stated
contract (`opts.fs` if yours differs).

Predictions and ground truth are paired by the **number in the filename**, not by directory
order. A prediction with no matching ground truth — or a ground truth with no submitted
prediction — is an error, never a silently skipped subject, since scoring a subset would
quietly change the average. Row ordering is checked rather than assumed: a file sorted
differently would still parse and would then be scored against the wrong samples, silently.

## The metrics

Every metric is computed **per subject first**, then aggregated, so a subject with more trials
or more switches never counts for more.

| | Definition | Aggregation over subjects |
|---|---|---|
| **Acc** | fraction of samples decoded correctly | mean |
| **SDT** | median switch detection time (s) | **median** |
| **L** | median of `exp(-SDT/tau0)`, `tau0 = 60 s` | mean |
| **H** | `2·Acc·L / (Acc + L)`, Acc as a **fraction** | from the aggregated Acc and L |

**SDT uses the median at both levels** because it is censored at both. An undetected switch
is `Inf`, so a subject who missed more than half of theirs has an infinite median; a mean
would let that one subject carry the whole cohort to `Inf`. The median tolerates it, and still
returns `Inf` when more than half the subjects are themselves infinite — a broken cohort
should read as broken rather than be quietly rescued. Acc and L keep the mean: both are
bounded (`L` lives in `[0,1]` because the exponential maps an infinite SDT to exactly 0), so
neither can be dragged away by one subject.

**H is harmonic** so that neither term can carry the other: a decoder that is accurate but
never notices a switch, or that reacts instantly but decodes at chance, lands near zero either
way, where an arithmetic mean would hand both a comfortable middling number. Acc must be a
fraction first — both terms have to share a scale or the harmonic mean collapses onto the
smaller one.

### How SDT is found

For each ground-truth switch, `computeSwitchLatency` fits the split point that best explains
the decoded sequence as *"old label, then new label"*, and scores how late that split is.
This is a different question from accuracy: a decoder can be wrong for twenty seconds after
every switch and still post a high accuracy if the segments are long.

1. Search the window `[t0, t0 + searchWindow]` — **forward only**, clipped to the trial. The
   decoder is causal, so it cannot turn over before the switch it has not yet seen evidence
   for; searching behind `t0` would only let noise fit a split early and score it as a fast
   detection.
2. Take the split `tau*` maximising agreement with the old label before it and the new label
   from it onward.
3. **Sustain check**: the detection counts only if at least `tSustain` of consecutive
   new-label predictions immediately follow `tau*`, so a single-sample flicker cannot be
   scored as a fast detection. On failure the search resumes just past `tau*`.
4. No split passes → the switch is **undetected**, `SDT = Inf`, so `L` contributes exactly 0
   rather than something imputed.

Constants live in `defaultLatencyOpts.m` — the single source of truth, since they define the
metric. Because a **median hides censoring below 50%**, always read the undetected count
beside SDT and L; a subject that missed a third of its switches otherwise looks like one that
caught them all a little late.

### What counts as a switch

A change of attended stream **within a trial**. A label change across a trial boundary is
not one — nothing switched between the end of one trial and the start of the next, and a
decoder reset at the boundary has no history it could have detected one from. Those changes
are excluded and counted separately.

**A dataset with no within-trial switches therefore has no SDT and no L, and they come back
`NaN`.** That is correct, not a failure, and Acc is unaffected. Part 1's
`generateSyntheticSwitches` exists to impose the missing structure — report anything it
produces as **synthetic**.

## The causality constraint

The predictions being scored are expected to obey:

> A decision for a sample may use only earlier data **from its own trial**, plus at most a
> configurable lookahead.

Scoring supports this rather than enforcing it — the CSV format cannot express when a decision
was made. Two consequences show up here:

* **Metrics are per sample, never per decision window.** Window-grid scoring would both credit
  a decision for the samples it was computed from and round every switch onto the window grid,
  capping SDT's resolution at the window length.
* **The latency search is clipped to the trial containing the switch**, so a split is never
  fitted against decisions another trial's model run produced.

Part 1 fills samples that no causal decision could cover with a single held random guess per
block, identical across decoders. Those samples are scored like any other, so a large guessed
fraction pulls Acc toward chance — Part 1 reports it as `nUndefined`.

## Output

```matlab
scores.Acc / .SDT / .L / .H     the aggregated metrics
scores.perSubject               per-subject values, plus raw per-switch .sdt / .ell
scores.nSwitches / .nUndetected / .nBoundaryChanges / .nSamples
scores.nSaturated               subjects whose own median SDT is Inf
scores.subjects
```

One directory is one test set. To combine several, score each and average the per-subject
values — `scores.perSubject` is there for exactly that.

## Files

| File | Role |
|---|---|
| **`crossDatasetScoring.m`** | **The main script. Start here — it is the only file you call.** |
| `scorePredictions.m` | Pair the files, score every subject, aggregate. |
| `computeSwitchLatency.m` | Fit the detection split per switch; SDT and L. |
| `readLabelCsv.m` | Read and validate one label file. |
| `reportScores.m` | Print the per-subject and headline tables. |
| `defaultLatencyOpts.m` | The latency constants (`searchWindow`, `tSustain`, `tUndetected`, `tau0`). |

## Citation

The HMM post-processing whose output this scores implements the method of:

> Heintz, N., Francart, T., & Bertrand, A. (2025). *Post-processing of EEG-based Auditory
> Attention Decoding Decisions via Hidden Markov Models.* arXiv:2506.24024.

**Please cite this paper if you use the HMM post-processing in your own work.**

Author: Simon Geirnaert, KU Leuven (Dept. of Electrical Engineering, ESAT).
