Invited Research Talk: Measuring Generalization in EEG Foundation Models

This talk presents a multi-dimensional evaluation and interpretability framework for EEG foundation models. It reveals that current models often fail to outperform supervised baselines for BCI tasks, lack robustness to sparse channels, and exhibit an aperiodic low-frequency bias, making them better at capturing subject-specific rather than task-specific information. The analysis highlights critical deficiencies and suggests future directions for pre-training objectives and data collection to improve generalization.

August 27, 2026 5 min read

Invited Research Talk: Measuring Generalization in EEG Foundation Models — source: https://www.youtube.com/watch?v=zl_RxpwPsIY

This talk by Adita, a fourth-year PhD student at USC, delves into the critical need for robust evaluation and interpretability frameworks for EEG (Electroencephalography) foundation models. Despite a surge in interest and proposed models over recent years, Adita highlights significant limitations in current evaluation practices, which often fail to assess true generalizability. These limitations include evaluation on single folds, use of entire fine-tuning datasets, and unfreezing foundation model backbones during downstream tasks, hindering true assessment of transfer learning capabilities.

Adita introduces a multi-dimensional evaluation framework focusing on parameter, sample, and channel efficiency to reflect real-world EEG deployment constraints. The framework was applied to six diverse datasets (motor imagery, mental health detection, sleep stage classification, epilepsy) and compared supervised baselines against three state-of-the-art EEG foundation models.

Key Findings from Multi-Dimensional Evaluation:

Downstream Performance: Contrary to expectations, EEG foundation models generally performed on par with or slightly worse than fully supervised baselines for Brain-Computer Interface (BCI) tasks. However, they showed modest performance gains in non-BCI tasks (e.g., sleep stage, mental health, epilepsy). This disparity is hypothesized to stem from a domain mismatch, as many foundation models are pre-trained on unlabeled hospital ICU data, which aligns more with non-BCI clinical scenarios.
Parameter Efficiency: When assessing the generalizability of learned embeddings (by comparing fully fine-tuned models with frozen-encoder models using LoRA or linear probing), foundation models exhibited lower parameter efficiency for BCI tasks but higher efficiency for non-BCI tasks. This suggests that for BCI, the embeddings are less generalizable, or full fine-tuning might be sub-optimal.
Sample Efficiency: With drastically reduced fine-tuning data (e.g., 50-200 samples), foundation models showed mixed results. They did not consistently outperform supervised models for BCI tasks but demonstrated better performance for sleep stage classification.
Channel Efficiency (Robustness to Missing Channels): A surprising finding was the lack of robustness of foundation models to missing or sparse channels. Despite pre-training on large datasets, they did not perform significantly better than supervised models when channels were restricted to specific lobes or sampled sparsely.

Interpretability Studies: Unveiling Model Biases

To understand why these models behave as they do, Adita’s team conducted interpretability studies, focusing on the two core components of EEG signals: aperiodic broadband spectra and oscillatory peaks. Aperiodic components typically relate to subject-specific information, while oscillatory components are often task-specific. Using a simulation-based framework, synthetic EEG signals with controllable components were generated and fed into foundation models to see which characteristics their embeddings captured.

Aperiodic Components: The models’ embeddings were highly sensitive and accurate in capturing aperiodic components (e.g., exponent, broadband offset) from the simulated data.
Oscillatory Components: In contrast, the models demonstrated insensitivity to oscillatory frequency (predictions were random with R-squared values near zero). Furthermore, while they could capture the relative power of oscillations, this capability decreased significantly for higher-frequency oscillations, indicating a bias towards lower-frequency oscillatory components.

This leads to the conclusion that EEG foundation models tend to have an aperiodic, low-frequency bias.

Subject vs. Task Specificity in Embeddings:

Building on the interpretability findings, an experiment was performed to determine if this bias translated into a preference for capturing subject-specific information over task-specific information. By training linear probes to predict either the downstream task or the subject ID from foundation model embeddings, it was found that performance for subject-specific prediction was consistently much higher (except for sleep stage, possibly due to its limited and specific channel configuration). This was further supported by consistently lower embedding distances for subject-specific clusters compared to task-specific clusters.

Open Questions and Future Directions:

The analysis underscores deficiencies in current EEG foundation models and points to critical areas for improvement:

Pre-training Objectives: The current reliance on mask reconstruction, effective in other modalities, might not be optimal for EEG due to its unique signal characteristics (e.g., low SNR, spectral properties). Exploring “top-down” objectives that incorporate spectral characteristics could be beneficial.
Data Collection and Domain: The mismatch between pre-training data (hospital ICUs) and downstream task domains (e.g., BCI) is a significant limitation. The need for better quality, real-world data with weak labels is paramount. Discussions also touched upon the difficulty of scaling models due to data scarcity and the band-limited nature of EEG, questioning whether massive parameter counts yield proportional gains. The potential of wearables to increase data availability was also noted.
Robustness: Addressing the lack of robustness to channel sparsity, perhaps through pre-training objectives that involve masking entire channels, is crucial.