We provide a brief step-by-step instruction for you to start the competition. **DISCLAIMER**: This page is for instructive purpose only. Please read the corresponding page(s) for detailed explanation. **DISCLAIMER**: Please ensure you fully understand the `Terms` page and other restrictions applied during your competition. **Attention**: For participants who plan to train a spatial sAAD decoder: Once you have downloaded the training datasets (which will be available once Development Phase begins), please keep in mind to convert the attended azimuth to a 0-based binary label system, with `0` representing attended to left, and `1` to the right. The attended azimuth is negative if attended to left. The baseline example code is not a spatial decoder, and therefore no conversion function is provided. **Attention**: We have made some changes on the websites, including the competition's `Terms`. Please go through the website to catch up the updates and understand the current rules. To get your training data: 1. Wait until Development Phase begin. 2. The Input Data and Scoring Program will be made available. 3. Download them, and unzip the file. 4. Find the link to download the full training data, and download them. 5. Unzip all files. To use the linear SR baseline (+HMM): 6. Start your MATLAB instance, change the path to the training data in the `runCrossDataasetMain.m` file. 7. Run the file. 8. Make other modification as you wish. To use the DNN baseline: 6. Use `conda`, `vene`, `uv`, `mamba` or any other Python environment management tools to create a new environment. Assume its name is `eegsaad`. 7. Activate the environment `conda activate eegsaad`. 8. Pull the [SuperHuge repo in Github](github.com/SeanZhang99/SuperHugeAAD). Use the `python-package-working` branch. 9. Pull the [MultidatasetAAD repo in Github](https://github.com/SeanZhang99/MultiDatasetAAD). Use the `main` branch. 10. Run the `CNDtoSuperhuge.py` to trasncribe the training dataset into the Superhuge format. 11. Change the path in data_config.yaml. 12. Run the respective python file to get started. **DISCLAIMER**: - The superhugeAAD and multidatasetAAD repos are in principle not required for this challenge. We provide these repos for the sake of convenience of teams who plan to develop a DNN model. - We do not guarantee these packages will work perfectly with the training data. - The package is developed as a unified pipeline and is suitable for experienced DNN users. - It is not mandatory to start from the provided baseline. You are free to use the dataset/datamodule in the repo and write your own training scripts. - We do not guarantee the models implemented in these packages are identical to the original versions in the literature. - If you use these repos in your work or refer to certain components of these repos, please give a **star** in the Github. ## Remember the 1second-lookahead restriction! See the 'Data & submissions' page. Note that this restriction does not prohibit you from using longer windows than 1 second. For example, when using 10s-windows, the 1s-lookahead can still be guaranteed: 1) for non-overlapping windows: by assigning the window's attention label only to the last second in the window, and to the first 9 seconds of the next window. 2) by using overlapping windows (for example, with shifts of < 1s), allowing to update the per-sample decision more frequently without violating the causality requirement. ## A reference results of the baseline codes ### Linear SR baseline and HMM post-processing (1s window) By running the Linear baseline on KUL2016, DTU, KUL-HD2021, USTC, and test the decoder on KUL-AVGC, with a decision window of 1 second, `lwcov` enabled, you will get: ``` Train KUL_AAD_UltraHD2021 30 subjects, 120 trials. Train DTU2018 18 subjects, 1080 trials. Train USTC2026 18 subjects, 360 trials. Train KUL2016 16 subjects, 320 trials. Training on 6975446 samples, 812 design columns (lwcov regularisation). Test AVGCAAD2024 13 subjects, 99 trials. ========== Cross-dataset ========== CV: train on KUL_AAD_UltraHD2021, DTU2018, USTC2026, KUL2016, test on AVGCAAD2024 (by dataset) Reg: lwcov bias: 0 Leeg: [0 0.4] s Subjects: 13 Subject 1s -------------------- 1 53.31 3 55.22 4 55.27 7 56.19 8 54.79 9 53.85 10 54.90 11 54.10 12 53.52 13 53.52 14 52.88 15 52.08 16 51.58 -------------------- Mean 53.94 Std 1.32 ``` By passing the output to the evaluation example code, you will get: (Some lines are omitted for clear demonstration) ``` ========== Scores: plain decoder ========== Acc (%) 53.89 SDT (s) 17.16 L 0.7583 H <- headline 0.6301 undetected 2 of 238 ========== Scores: causal HMM ========== Acc (%) 68.4 SDT (s) 18.53 L 0.7173 H 0.7002 undetected 16 of 238 ========== Headline comparison ========== Acc (%) SDT (s) L H plain 53.89 17.16 0.7583 0.6301 causal HMM 68.4 18.53 0.7173 0.7002 ``` For a decision window with 50 seconds (hop-size=0.6s, no HMM applied), you will get ``` -------------------- Mean 73.20 Std 7.44 ``` when you run `crossDatasetMain.m`. The results may be different if you pass the label sequence into the evaluation function and perform a per-time-point (per-sample, in our context) computation. ### VLAAI SR baseline without post-processing (30s window) By training a VLAAI model with the default hyperparameters and optimizers, you will get ``` ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ test/acc_by_pcc │ 0.5136363506317139 │ │ test/loss │ -0.003495404263958335 │ └───────────────────────────┴────────────────────────────┘ ``` with a decision window of **30 seconds**. We note that the decoding accuracy obtained by the provided DNN model is substantially lower than that of the linear SR decoder, and the results vary widely when different datasets are used as the test set. Generalization to unseen data is known to be a major challenge for DNN-based AAD models. You may wish to further investigate the reason for this low decoding accuracy and try other approaches or architectures to improve decoding performance on unseen datasets, while making sure to follow the competition rules. **DISCLAIMER**: The DNN baseline may violate the causality constraint because of the default kernel size and the number of CNN layers. Please be careful when developing your own model. You may use a model interface that is `with envelope dumped` (the superhuge baseline includes one, though it is not enabled by default) to save the test-time predictions and compute per-sample scores later. Alternatively, you can write your own test scripts for this purpose.