Friedman Nemenyi Eeg Bci Benchmark

---
name: friedman-nemenyi-eeg-bci-benchmark
description: Statistical benchmarking methodology for EEG motor-imagery BCI decoders using Friedman-Nemenyi tests. Proves no single decoding pipeline dominates across subjects — personalized model selection adds ~7% accuracy over best fixed choice. Use when evaluating BCI decoders, comparing multi-classifier performance, or designing subject-aware BCI systems.
activation: EEG BCI benchmark, motor imagery decoder comparison, Friedman-Nemenyi, per-subject optimality, BCI personalization, MOABB, CSP, covariance tangent-space
tags: [neuroscience, BCI, EEG, benchmarking, statistical-testing, motor-imagery, personalization]
version: 1.0.0
author: agent
arxiv_id: "2606.24394"
---

# Friedman-Nemenyi EEG Motor-Imagery BCI Benchmark

## Core Contribution

Rigorous statistical framework proving that **average rankings mask per-subject optimality** in EEG motor-imagery BCI decoding. The "best" pipeline on average is optimal for only 35% of participants — personalized model selection adds ~7 accuracy points.

## Key Findings

### Experimental Scale
- **1,056 decoding configurations** (feature extractor × scaler × classifier)
- **>340,000 subject-level model fits**
- **3 public datasets**: PhysionetMI (109 participants), Cho2017 (52), Zhou2016 (4)
- **2 frequency bands**: 8-15 Hz, 8-30 Hz
- Every model fit and tested within single session of single participant (easiest regime)

### Statistical Analysis
- **Friedman omnibus tests** for overall significance
- **Nemenyi critical-difference analysis** for pairwise comparisons
- **Wilcoxon signed-rank tests** with effect sizes
- Results on PhysionetMI (largest cohort): cov-tgsp and CSP families are strongest but **statistically indistinguishable** (Nemenyi p=0.27; Kendall's W=0.11)

### Central Result: Personalization Problem
- Single best pipeline optimal for only **35% of PhysionetMI participants**
- Nonlinear descriptors best for roughly **1/3 of participants**
- **Matching pipeline to participant adds ~7 accuracy points** over best fixed choice
- Ranking not artifact of dimensionality
- Classifier and scaler choices **secondary to feature representation**

## Methodology

### Multi-Classifier Comparison Statistics
```
1. Friedman test (non-parametric repeated measures ANOVA)
   → Tests if any classifier significantly differs from others
   
2. Nemenyi post-hoc test (critical difference diagram)
   → Pairwise comparisons with family-wise error correction
   → If p > 0.05: pipelines are statistically equivalent
   
3. Wilcoxon signed-rank + effect sizes
   → Individual pairwise comparisons with magnitude
```

### Feature Extractor Families
- **Covariance tangent-space projection (cov-tgsp)**: Riemannian geometry approach
- **Common Spatial Patterns (CSP)**: Classic spatial filtering
- **Nonlinear descriptors**: Capture complex feature interactions
- Ordering is **dataset-dependent** — no universal winner

### Framework
- **MOABB** (Mother of All BCI Benchmarks) for reproducible evaluation
- Within-subject, within-session cross-validation

## Practical Implications

1. **No universal BCI decoder exists** — even in easiest regime
2. **Personalized model selection is essential** — 7% accuracy gain is clinically significant
3. **Feature representation > classifier choice** — invest in feature engineering
4. **Average rankings are misleading** — always report per-subject distributions
5. **Nonlinear methods deserve attention** — optimal for ~33% of users

## Pitfalls

1. **Don't claim one pipeline is "best"** — statistically indistinguishable on large cohorts
2. **Don't ignore per-subject variance** — average accuracy hides individual differences
3. **Don't compare only on one dataset** — ordering is dataset-dependent
4. **Don't skip effect sizes** — statistical significance ≠ practical significance
5. **Don't ignore nonlinear methods** — they're optimal for 1/3 of users

## Related Skills

- `cross-subject-eeg-decoding` — complementary cross-subject generalization approaches
- `eeg-preprocessing-reliability` — preprocessing impacts on decoding reliability
- `ferroelectric-snn-eeg` — neuromorphic EEG BCI personalization
- `eeg-mftnet-multi-scale-temporal` — specific decoder architecture
- `bci-sift-feature-selection` — systematic feature tuning

## Source

Vasques, X., Barbaste, P., & Oullier, O. (2026). Average Rankings Mask Per-Subject Optimality: A Friedman-Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders. arXiv:2606.24394.