## Evaluation The three ranking-relevant scores (Accuracy, Switch Detection Latency Score, Harmonic Mean) are **normalized scores in [0, 1], the larger the better**. Chance level is 0.5 for accuracy. In addition, a raw **Switch Detection Time reference column** (in seconds, the smaller the better) is shown in the leaderboard, for reference only, and does not affect the ranking — see Section 5.2. ### Accuracy (Acc) For each test set, the accuracy is computed as the average, over all subjects in that test set, of each subject's own accuracy: ![](https://img.cdn1.vip/i/6a92894006a83_1787988288.jpeg) where: - **N** = number of subjects in the test set - **s** = a subject index (each subject counts equally, no matter how many trials/samples) - **Cₛ** = number of correctly decoded samples of subject s - **Tₛ** = total number of samples of subject s Accuracy is computed separately for each test set, and the test sets are combined with **equal weights** (i.e., the final Acc is the arithmetic average of the per-test-set accuracies). ### Switch Detection Latency Score (L) For each ground-truth attention switch, we measure how quickly the predicted label sequence flips to the new label: - **Search window**: for a switch occurring at time **t₀**, we consider the window [t₀ s, t₀ + 100 s]. - **Optimal split τ\***: inside the window we choose the split point **τ\*** (tau-star, in seconds) that maximizes the number of "agreements": samples before τ\* are compared with the old actual attended label, samples from τ\* onward are compared with the new actual attended label. τ\* is the point that maximizes the sum of correct prediction across the search window ![](https://img.cdn1.vip/i/6a92894348fe0_1787988291.jpeg) - **Validity**: the detection is valid only if at least **t_sustain = 0.5 s** (64 samples at 128 Hz) of **consecutive** predictions equal to the new label immediately follow τ\*. In case the sustain validation is not passed, the algorithm continues searching for a next τ from the current τ\*. If no switch is detected, the τ\* is set to **+Inf**. - **Switch detection time**: **SDTₖ = abs(τ\* − t₀)** in seconds for detected switch k. - **Per-switch score**: **ℓₖ = exp(−SDTₖ / 60)** in (0, 1], where 60 s is a time constant T₀. A switch detected instantly gets 1.0; a switch detected 60 s late gets ≈ 0.37; **undetected switch score is 0 (normalized from + Inf)**. - **Latency score L**: For each subject, the median of per-switch scores is used. The latency score L is finally mean-averaged across all subjects to provide the final evaluation switch detection latency score, ranging from 0 to 1, the higher the better. **Reference column (raw SDT, in seconds)**: for reference only, the leaderboard additionally shows the raw median switch detection time in seconds. It is the median value of SDT across all switches per subject, then the median across subjects; **undetected switches are counted as +Inf and handled by the median**. This column does **not** affect the ranking. **We note that the calculation of raw SDT (median+median) is not the same as the normalized SDT (median+mean). ** ### Final Ranking — Harmonic Mean (H) The final ranking is based on the **harmonic mean** of accuracy and latency: **H = 2 × Acc × L / (Acc + L)**