Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning

# Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning
Source: [https://arxiv.org/html/2609.17886](https://arxiv.org/html/2609.17886)
###### Abstract

EEG decoding tasks can rely on different temporal dynamics and cross\-channel relationships\. We test whether specialized modules improve a fully fine\-tuned EEG foundation model by augmenting CBraMod with cross\-depth Attention Residuals \(AttnRes\) and two soft\-routed expert banks\. Across matched three\-seed experiments on FACED, ISRUC, SEED\-V, and PhysioNet\-MI, the complete model changes mean balanced accuracy relative to full fine\-tuning by−0\.12\-0\.12,\+1\.27\+1\.27,\+0\.77\+0\.77, and−1\.27\-1\.27points, respectively\. AttnRes alone improves mean balanced accuracy on three datasets, whereas adding experts on top of AttnRes helps only FACED and SEED\-V\. These gains come with substantial overhead: AttnRes requires 2\.11–2\.88×\\timesruntime and 1\.78–2\.67×\\timesmemory, while the complete model requires 2\.41–3\.04×\\timesruntime and 1\.86–2\.85×\\timesmemory\. Overall, the added modules produce dataset\-dependent, sometimes opposing effects rather than consistent gains over full fine\-tuning\.

###### Index Terms:

EEG foundation models, CBraMod, fine\-tuning, soft\-routed expert capacity, EEG adaptation

††address:1Department of Computer Science, Vanderbilt University
2Department of Biomedical Informatics and Data Science, Yale University††footnotetext:This work has been submitted to the IEEE for possible publication\. Copyright may be transferred without notice, after which this version may no longer be accessible\.## 1Introduction

Electroencephalography \(EEG\) records multichannel neural activity with high temporal resolution and supports diverse decoding tasks, including emotion recognition, sleep staging, and motor imagery\. Decoding these signals requires modeling patterns distributed across time and channels, but the relevant structure can differ across tasks and datasets\. Conventional EEG models therefore often include explicit temporal and spatial processing\[[7](https://arxiv.org/html/2609.17886#bib.bib6),[12](https://arxiv.org/html/2609.17886#bib.bib7)\]\.

Recent EEG foundation models pretrain large encoders on broad EEG corpora and transfer their representations across downstream tasks\. Models such as CBraMod, LaBraM, and BrainGPT demonstrate this paradigm for transferable EEG representation learning\[[13](https://arxiv.org/html/2609.17886#bib.bib1),[4](https://arxiv.org/html/2609.17886#bib.bib2),[14](https://arxiv.org/html/2609.17886#bib.bib3)\]\. When adapting such a pretrained encoder, full fine\-tuning allows all backbone parameters to change\. This raises a practical question: do additional specialized modules still help, or can a fully trainable backbone learn similar transformations on its own?

We investigate this question with CBraMod using two complementary mechanisms motivated by the variability of EEG decoding\. First, cross\-depth Attention Residuals \(AttnRes\) combine representations from earlier encoder layers, allowing downstream processing to draw on features learned at different depths rather than relying only on the standard residual path\. This may be useful when different EEG tasks depend on different temporal or cross\-channel patterns\. Second, soft\-routed expert banks use input\-dependent routing weights to combine several nonlinear expert networks, allowing the model to apply different transformations to different EEG inputs\. Related expert\-routing methods have recently been explored for EEG representation learning\[[9](https://arxiv.org/html/2609.17886#bib.bib4)\]\. We therefore compare full fine\-tuning, full fine\-tuning with AttnRes, and the complete model with AttnRes and experts to test whether these mechanisms provide complementary benefit beyond a fully trainable backbone\.

Our contributions are threefold\. First, we test whether these additional modules provide further benefit when a pretrained EEG backbone is already fully trainable\. Second, we separate the effects of cross\-depth aggregation and soft\-routed expert capacity using matched three\-seed comparisons across emotion recognition, sleep staging, and motor imagery\. Third, we characterize dataset\-dependent performance changes together with computational cost and class\- and subject\-level effects\.

Figure 1:Cross\-depth expert augmentation and staged comparisons\. Cross\-depth AttnRes combines earlier CBraMod states before the final representation is processed by two separate soft\-routed expert banks in parallel with the shared FFN\. The lower panel summarizes the evaluated stages: Full FT, \+AttnRes, and \+AttnRes \+ Experts\.
## 2Cross\-Depth and Soft\-Routed Expert Capacity

Our baseline fully fine\-tunes CBraMod\. In the augmented models, the backbone remains trainable while we add cross\-depth Attention Residuals \(AttnRes\) and two separate expert banks\. AttnRes learns a weighted combination of earlier block outputs instead of using only the standard residual path\[[6](https://arxiv.org/html/2609.17886#bib.bib5)\]\.

### 2\.1Cross\-depth aggregation

CBraMod processes the EEG representation through a stack of encoder blocks\. For target blockii, letHj∈𝐑B×C×S×DH\_\{j\}\\in\\mathbf\{R\}^\{B\\times C\\times S\\times D\}denote an earlier representation that can be used as a source, and let𝒮i\\mathcal\{S\}\_\{i\}denote the available sources, including the patch embedding and outputs of preceding blocks\. We form

αi,j\\displaystyle\\alpha\_\{i,j\}=softmaxj∈𝒮i⁡\(qi⊤​RMSNormi⁡\(Hj\)\),\\displaystyle=\\operatorname\{softmax\}\_\{j\\in\\mathcal\{S\}\_\{i\}\}\\\!\\left\(q\_\{i\}^\{\\top\}\\operatorname\{RMSNorm\}\_\{i\}\(H\_\{j\}\)\\right\),\(1\)H~i\\displaystyle\\widetilde\{H\}\_\{i\}=∑j∈𝒮iαi,j​Hj,\\displaystyle=\\sum\_\{j\\in\\mathcal\{S\}\_\{i\}\}\\alpha\_\{i,j\}H\_\{j\},Hereqi∈𝐑Dq\_\{i\}\\in\\mathbf\{R\}^\{D\}andRMSNormi\\operatorname\{RMSNorm\}\_\{i\}are learned per target block \(400 parameters per block, 4,800 across 12 blocks\)\. In the ungated setting,H~i\\widetilde\{H\}\_\{i\}replaces the ordinary input beforenorm1\\operatorname\{norm1\}and self\-attention, which then applies its usual residual update\. The softmax weights are computed across the available depth states at each position, and aggregation begins at encoder block 0 rather than only at final pooling\.

### 2\.2Soft\-routed expert capacity

At the top \(final\) encoder layer, letBBdenote the baseline pre\-FFN representation andAAits AttnRes\-augmented counterpart\. We form one routing feature vector per sample,u=\[μ⁡\(B\);μ⁡\(A\);μ⁡\(A−B\)\]u=\[\\mu\(B\);\\mu\(A\);\\mu\(A\-B\)\], whereμ\\muaverages the channel and patch axes\. The implemented router uses all three terms, including the differenceA−BA\-B\. LetXXbe the layer\-normalized pre\-FFN representation supplied to the experts\. Each expert is a 200–800–200 GELU MLP using the dropout rate specified by the dataset protocol\. For each bankb∈\{A,B\}b\\in\\\{A,B\\\}, the model computes

p\(b\)\\displaystyle p^\{\(b\)\}=softmax\(rb\(u\)/τ\),Rb\(X\)=∑e=14p\(b\)eE\(b\)e\(X\),\\displaystyle=\\operatorname\{softmax\}\(r\_\{b\}\(u\)/\\tau\),\\hskip 18\.49988ptR\_\{b\}\(X\)=\\sum\_\{e=1\}^\{4\}p^\{\(b\)\}\_\{e\}E^\{\(b\)\}\_\{e\}\(X\),\(2\)F⁡\(X\)\\displaystyle F\(X\)=Fshared​\(X\)\+RA​\(X\)\+RB​\(X\),\\displaystyle=F\_\{\\rm shared\}\(X\)\+R\_\{A\}\(X\)\+R\_\{B\}\(X\),withτ=1\.5\\tau=1\.5\. HereFsharedF\_\{\\rm shared\}is the ordinary dense FFN output\. The two banks use separate routers and expert parameters but receive the same normalized token representation\. They are separate learned parameter banks, not fixed spatial or spectral operators\. Each routing vector is computed once per sample by averaging channel and patch axes and is shared across that sample’s tokens\. The router forms a weighted mixture of all four experts in each bank\. We use dense routing, so every expert contributes to each sample according to its routing weight\. This avoids additional effects from sparse expert dispatch or load\-balancing mechanisms\[[11](https://arxiv.org/html/2609.17886#bib.bib8),[2](https://arxiv.org/html/2609.17886#bib.bib9)\]\.

## 3Experimental Protocol

We evaluate FACED and SEED\-V for emotion recognition, ISRUC for sleep staging, and PhysioNet\-MI for motor imagery\[[1](https://arxiv.org/html/2609.17886#bib.bib10),[8](https://arxiv.org/html/2609.17886#bib.bib11),[5](https://arxiv.org/html/2609.17886#bib.bib12),[10](https://arxiv.org/html/2609.17886#bib.bib13),[3](https://arxiv.org/html/2609.17886#bib.bib14)\]\. Table[1](https://arxiv.org/html/2609.17886#S3.T1)summarizes the fixed dataset splits and training schedules\. Within each pair, both conditions use the same pretrained CBraMod checkpoint, preprocessing, split, classifier, and checkpoint\-selection rule\. For ISRUC, we use the same ordered split of subjects 1–80/81–90/91–100, and it is unchanged across paired conditions\. SEED\-V retains the fixed CBraMod LMDB benchmark protocol for comparability and is not subject\-disjoint\.

Table 1:Dataset and training protocols used in the matched comparisons\. Ch\., EEG channels; BS, batch size\.∗SEED\-V uses the fixed CBraMod LMDB benchmark split rather than a subject\-disjoint partition\.Each condition is run with three prespecified random seeds, 3407, 2024, and 2027\. The best checkpoint is selected using validation Cohen’sκ\\kappaand then evaluated once on the held\-out test split\. We report balanced accuracy \(BA\), weighted\-F1 \(wF1\), and Cohen’sκ\\kappa\. Scores in the result tables are scaled by 100 for compact presentation, and all deltas are computed from paired runs that use the same dataset protocol and seed\. Because we use only three prespecified seeds, we treat the paired differences as descriptive rather than inferential\.

*Implementation details\.*We use ungated pre\-attention AttnRes from layer 0\. Routers use a hidden width of 128, and expert output weights are zero\-initialized\. All parameters follow the fixed dataset schedules in Table[1](https://arxiv.org/html/2609.17886#S3.T1); the classifier uses all patch representations\.

End\-to\-end training wall\-clock time and peak allocated CUDA memory are measured in the same NVIDIA A6000 environment for matched runs\. For diagnostics, class\-wise recall is averaged over matched seeds\. For PhysioNet\-MI, we additionally analyze paired BA changes across the 17 held\-out subjects and estimate uncertainty by bootstrapping those subjects\. Validation trajectories use the checkpoint\-selection criterion\.

## 4Results and Discussion

Tables[3](https://arxiv.org/html/2609.17886#S4.T3)and[3](https://arxiv.org/html/2609.17886#S4.T3)summarize test performance, paired augmentation effects, and computational overhead across the four datasets\.

![Refer to caption](https://arxiv.org/html/2609.17886v1/figure2_classwise_deltas.png)Figure 2:Mean changes in class\-wise recall over three matched seeds\. The first row shows AttnRes minus Full FT and the second shows the complete model minus AttnRes; values are descriptive percentage\-point changes, and the shared diverging color scale is centered at zero and spans−12\-12to\+12\+12\.Table 2:Test performance under matched full fine\-tuning and cross\-depth expert augmentation\. Each method cell reports BA / wF1 /κ\\kappaas mean±\\pmsample SD over seeds 3407, 2024, and 2027\. All scores are multiplied by 100 for compact presentation; Full FT serves as the within\-protocol reference for each dataset\.Table 3:Staged component comparison and cost\. BA is mean±\\pmSD over the matched seeds and is scaled by 100\.Δ\\DeltaAttnRes denotes AttnRes−\-Full FT, andΔ\\DeltaExperts∣\\midAttnRes denotes the complete model−\-AttnRes\. The time and memory columns \(C denotes the complete model\) are ratios to Full FT\.### 4\.1Heterogeneous performance shifts

The complete model shows heterogeneous, dataset\-dependent effects\. ISRUC shows the largest positive complete\-model mean BA difference: all three paired seeds favor the complete model, with mean BA higher by1\.27±1\.461\.27\\pm 1\.46points; weighted\-F1 andκ\\kappaincrease by1\.90±1\.661\.90\\pm 1\.66and2\.40±1\.892\.40\\pm 1\.89points\. PhysioNet\-MI shows the opposite BA shift,−1\.27±0\.26\-1\.27\\pm 0\.26points, with all seeds negative\. FACED does not show a consistent advantage across the three matched seeds \(−0\.12±1\.24\-0\.12\\pm 1\.24BA; one positive seed\), whereas SEED\-V has a modestly higher mean \(\+0\.77±0\.74\+0\.77\\pm 0\.74BA; two positive seeds\)\. Across all four datasets, BA, weighted\-F1, andκ\\kappashow the same direction of change\.

The best\-performing stage therefore varies by dataset \(Table[3](https://arxiv.org/html/2609.17886#S4.T3)\)\.

### 4\.2Component\-wise incremental effects

Table[3](https://arxiv.org/html/2609.17886#S4.T3)definesΔAttnRes\\Delta\_\{\\rm AttnRes\}as AttnRes minus Full FT andΔExperts\|AttnRes\\Delta\_\{\\rm Experts\\mid AttnRes\}as the complete model minus AttnRes\. AttnRes raises mean BA relative to Full FT on ISRUC, SEED\-V, and PhysioNet\-MI, but lowers it on FACED\. Adding experts on top of AttnRes decreases mean BA on ISRUC and PhysioNet\-MI but increases it on FACED and SEED\-V; weighted\-F1 andκ\\kappashow the same directions\. These opposing stage effects show that comparing only Full FT with the final model can hide how AttnRes and the expert module contribute differently\. Because adding experts also increases parameter count,ΔExperts\|AttnRes\\Delta\_\{\\rm Experts\\mid AttnRes\}measures the overall incremental effect of the full expert module rather than isolating routing alone\.

At the selected checkpoints, the entropy\-based effective expert count, which summarizes how many experts are meaningfully used, ranged from 2\.57–4\.00 across datasets, seeds, and banks, while the dominant expert received up to 0\.91 of top\-1 assignments\. These inference\-only router statistics were not used for model selection\.

### 4\.3Class\-wise stage effects

To characterize which classes contribute to the aggregate changes, we re\-evaluated the selected test checkpoints and computed class\-wise recall changes between matched conditions\. Figure[2](https://arxiv.org/html/2609.17886#S4.F2)shows concentrated, dataset\-specific class effects\. On ISRUC, AttnRes mainly increases N2 recall \(\+11\.7 points\) while reducing N3 \(−7\.2\-7\.2\), and the expert stage partially reverses both changes\. On PhysioNet\-MI, experts reverse the AttnRes gain for class 2 and reduce recall for classes 0–1\. These patterns show that different classes gain and lose performance at different stages rather than all classes changing uniformly\.

On PhysioNet\-MI, experts reduce BA for 12/17 held\-out subjects \(median pairedΔ\\DeltaBA=−1\.98=\-1\.98; subject\-bootstrap 95% CI\[−3\.29,−0\.16\]\[\-3\.29,\-0\.16\]\)\. On ISRUC, AttnRes remains above Full FT in validationκ\\kappafor all seeds across epochs 19–23 \(\+4\.1 to \+5\.2 mean points\), whereas the PhysioNet\-MI expert disadvantage is less consistent on validation\.

### 4\.4Accuracy–compute trade\-off

These accuracy changes incur substantial compute overhead\. AttnRes adds only 4,800 parameters but incurs 2\.11×\\times–2\.88×\\timesruntime and 1\.78×\\times–2\.67×\\timesmemory; the complete model adds 2\.73M parameters and incurs 2\.41×\\times–3\.04×\\timesruntime and 1\.86×\\times–2\.85×\\timesmemory\. Because AttnRes retains intermediate activations, parameter count alone does not capture its practical compute cost\.

## 5Limitations and Conclusion

This study is limited to one backbone, one protocol per dataset, and three seeds; SEED\-V is not subject\-disjoint, and dataset differences conflate task and split\. The expert\-stage comparison lacks a parameter\-matched dense control\. In addition, ungated AttnRes changes the block input from the start of fine\-tuning, and the diagnostic analyses are descriptive\.

Across the evaluated CBraMod settings, cross\-depth aggregation and expert capacity show dataset\-dependent, sometimes opposing effects rather than consistent gains over full fine\-tuning\. Thus, each added component should be evaluated separately rather than only through a baseline\-to\-complete comparison\.

## Compliance with Ethical Standards

This study is a secondary analysis of previously collected EEG data from publicly available datasets\. No new participants were recruited or contacted, and no new participant\-facing procedures were performed\. The present analysis used only previously collected, publicly available data and therefore did not require additional ethical approval; ethical approval and informed\-consent procedures for the original data collections are documented in the corresponding dataset publications and applicable access terms\.

## Funding and Conflicts of Interest

We thank the Vanderbilt Institute for Surgery and Engineering \(VISE\) Summer Research Program for funding support\. The authors declare no conflicts of interest\.

## References

- \[1\]J\. Chen, X\. Wang, C\. Huang, X\. Hu, X\. Shen, and D\. Zhang\(2023\)A large finer\-grained affective computing eeg dataset\.Scientific Data10\(1\)\.External Links:[Document](https://dx.doi.org/10.1038/s41597-023-02650-w)Cited by:[§3](https://arxiv.org/html/2609.17886#S3.p1.1)\.
- \[2\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2\.2](https://arxiv.org/html/2609.17886#S2.SS2.p1.3)\.
- \[3\]A\. L\. Goldberger, L\. A\. N\. Amaral, L\. Glass, J\. M\. Hausdorff, P\. Ch\. Ivanov, R\. G\. Mark, J\. E\. Mietus, G\. B\. Moody, C\. Peng, and H\. E\. Stanley\(2000\)PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals\.Circulation101\(23\),pp\. e215–e220\.External Links:[Document](https://dx.doi.org/10.1161/01.CIR.101.23.e215)Cited by:[§3](https://arxiv.org/html/2609.17886#S3.p1.1)\.
- \[4\]W\. Jiang, L\. Zhao, and B\. Lu\(2024\)Large brain model for learning generic representations with tremendous eeg data in bci\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p2.1)\.
- \[5\]S\. Khalighi, T\. Sousa, J\. M\. d\. Santos, and U\. Nunes\(2016\)ISRUC\-sleep: a comprehensive public dataset for sleep researchers\.Computer Methods and Programs in Biomedicine124,pp\. 180–192\.External Links:[Document](https://dx.doi.org/10.1016/j.cmpb.2015.10.013)Cited by:[§3](https://arxiv.org/html/2609.17886#S3.p1.1)\.
- \[6\]Kimi Team\(2026\)Attention residuals\.arXiv preprint arXiv:2603\.15031\.Cited by:[§2](https://arxiv.org/html/2609.17886#S2.p1.1)\.
- \[7\]V\. J\. Lawhern, A\. J\. Solon, N\. R\. Waytowich, S\. M\. Gordon, C\. P\. Hung, and B\. J\. Lance\(2018\)EEGNet: a compact convolutional neural network for eeg\-based brain\-computer interfaces\.Journal of Neural Engineering15\(5\),pp\. 056013\.External Links:[Document](https://dx.doi.org/10.1088/1741-2552/aace8c)Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p1.1)\.
- \[8\]W\. Liu, J\. Qiu, W\. Zheng, and B\. Lu\(2022\)Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition\.IEEE Transactions on Cognitive and Developmental Systems14\(2\),pp\. 715–729\.External Links:[Document](https://dx.doi.org/10.1109/TCDS.2021.3071170)Cited by:[§3](https://arxiv.org/html/2609.17886#S3.p1.1)\.
- \[9\]F\. Ma, Q\. An, P\. Chen, L\. Qian, X\. Lan, M\. Jiang, Z\. Gu, X\. Papademetris, and H\. Xu\(2026\)TRACE: temporal routing with autoregressive cross\-channel experts for eeg representation learning\.arXiv preprint arXiv:2605\.11380\.Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p3.1)\.
- \[10\]G\. Schalk\(2009\)EEG motor movement/imagery dataset\.PhysioNet\.External Links:[Document](https://dx.doi.org/10.13026/C28G6P)Cited by:[§3](https://arxiv.org/html/2609.17886#S3.p1.1)\.
- \[11\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. V\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2609.17886#S2.SS2.p1.3)\.
- \[12\]Y\. Song, Q\. Zheng, B\. Liu, and X\. Gao\(2023\)EEG conformer: convolutional transformer for eeg decoding and visualization\.IEEE Transactions on Neural Systems and Rehabilitation Engineering31,pp\. 710–719\.External Links:[Document](https://dx.doi.org/10.1109/TNSRE.2022.3230250)Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p1.1)\.
- \[13\]J\. Wang, S\. Zhao, Z\. Luo, Y\. Zhou, H\. Jiang, S\. Li, T\. Li, and G\. Pan\(2025\)CBraMod: a criss\-cross brain foundation model for eeg decoding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p2.1)\.
- \[14\]T\. Yue, X\. Gao, S\. Xue, Y\. Tang, L\. Guo, J\. Jiang, and J\. Liu\(2024\)BrainGPT: unleashing the potential of eeg generalist foundation model by autoregressive pre\-training\.arXiv preprint arXiv:2410\.19779\.Cited by:[§1](https://arxiv.org/html/2609.17886#S1.p2.1)\.