Neuralbench Unified Neuroai Benchmark

---
name: neuralbench-unified-neuroai-benchmark
description: "NeuralBench unified benchmarking framework for NeuroAI models. Standardized evaluation across EEG/MEG/fMRI tasks with 36 tasks, 14 architectures, 94 datasets. Covers foundation model evaluation, task-specific baselines, cross-modal extension. Activation: neuralbench, neuroai benchmark, brain model evaluation, EEG benchmark, fMRI benchmark, MEG benchmark, NeuroAI evaluation."
---

# NeuralBench: Unified Framework to Benchmark NeuroAI Models

> A standardized, extensible benchmarking framework for AI models processing brain recordings, with EEG v1.0 covering 36 tasks, 14 architectures, and 94 datasets.

## Metadata
- **Source**: arXiv:2605.08495
- **Authors**: Hubert Banville, Stéphane d'Ascoli, Simon Dahan, Jérémy Rapin, Marlène Careil, Yohann Benchetrit, Jarod Lévy, Saarang Panchavati, Antoine Ratouchniak, Mingfang (Lucy) Zhang, Elisa Cascardi, Katelyn Begany, Teon Brooks, Jean-Rémi King
- **Published**: 2026-05-08
- **Pages**: 31 pages, 9 figures

## Core Methodology

### Key Innovation
NeuralBench addresses the fragmentation in NeuroAI model evaluation by providing:
1. **Standardized interface** for accessing brain recording datasets
2. **Unified task definitions** across preprocessing, training, and evaluation
3. **Cross-modal extensibility** designed from the start for EEG, MEG, fMRI

### NeuralBench-EEG v1.0
- **36 electroencephalography (EEG) tasks** spanning cognitive decoding, clinical predictions, and more
- **14 deep learning architectures** evaluated systematically
- **94 datasets** accessed through a standardized interface
- Open-source framework inviting community expansion

### Key Findings
1. **Foundation models only marginally outperform task-specific models** in current benchmarks
2. **Many tasks remain highly challenging** (cognitive decoding, clinical predictions) even for the best models
3. **Preliminary extensions** to MEG and fMRI demonstrate cross-modal capability

## Technical Framework

### Architecture
```
NeuralBench Framework
├── Dataset Interface (standardized access)
├── Task Definitions (preprocessing + training + evaluation)
├── Model Registry (architecture implementations)
├── Evaluation Pipeline (metrics, benchmarks)
└── Extension API (new tasks/datasets/modalities)
```

### Benchmarking Dimensions
- **Task diversity**: Motor imagery, cognitive state decoding, clinical prediction, etc.
- **Model diversity**: CNNs, Transformers, foundation models, task-specific models
- **Dataset scale**: 94 datasets for EEG alone
- **Metric standardization**: Unified evaluation across all task-dataset-model combinations

## Applications
- **NeuroAI model comparison**: Systematic benchmarking of brain-AI models
- **Foundation model evaluation**: Testing generalization across brain tasks
- **Clinical AI validation**: Assessing model performance on clinical prediction tasks
- **Cross-modal research**: Extending from EEG to MEG/fMRI within same framework
- **Research direction identification**: Finding gaps where models underperform

## Implementation Guide

### Prerequisites
- Python environment with PyTorch
- Access to EEG/MEG/fMRI datasets (framework provides standardized interface)

### Steps
1. Install NeuralBench framework (open-source)
2. Select target modality (EEG, MEG, fMRI)
3. Choose tasks from available task registry
4. Register or implement model architecture
5. Run standardized evaluation pipeline
6. Compare results across models and tasks

### Extending the Framework
- Add new tasks via task definition API
- Add new datasets via standardized dataset interface
- Add new models via model registry
- Add new modalities (MEG, fMRI) following existing pattern

## Pitfalls
- Foundation models show only marginal gains over task-specific models currently
- Many clinically-relevant tasks remain unsolved even for best models
- Benchmark coverage is EEG-first; MEG/fMRI extensions are preliminary
- Preprocessing variations across studies can affect comparability

## Related Skills
- ai-science-benchmarking
- open-ended-science-benchmark
- eeg-foundation-model-adapters
- tta-eeg-foundation-models
- brain-dit-fmri-foundation-model