Eeg Based Lm Evaluation

---
name: eeg-based-lm-evaluation
description: Skill for evaluating language models using EEG signals to examine human-like next-word prediction behavior based on arXiv:2607.16549. Enables fine-grained analysis of cognitive plausibility of language models during reading comprehension tasks.
---
# EEG-Based Language Model Evaluation

## Overview
This skill encapsulates the methodology from arXiv:2607.16549, which proposes using EEG-recorded event-related potentials (ERPs) to evaluate the cognitive plausibility of language models during reading comprehension. The approach generates regressors based on information measures (top-1 prediction and surprisal) to predict ERP components, enabling fine-grained analysis of how well language models capture human linguistic processing.

## Core Concepts
- **Next-word predictability**: Both humans and language models exhibit predictability in word sequences during reading
- **ERP as cognitive signal**: Event-related potentials from EEG reflect different stages of cognitive processing
- **Information measures**: Top-1 prediction accuracy and surprisal (negative log probability) as predictors of ERP components
- **Surprisal correlation**: Surprisal potentially correlates with language-processing ERPs, especially for open-class words with high semantic content
- **Scaling limitations**: Increased model parameters and computational budgets do not consistently improve convergence with human-like linguistic processing

## Implementation Steps
1. **Data Collection**
   - Record EEG during reading comprehension tasks with millisecond resolution
   - Present identical text stimuli to human participants and language models
   - Extract event-related potentials (ERPs) time-locked to word onset

2. **Language Model Processing**
   - Generate probability distributions over vocabulary for each word context
   - Calculate top-1 prediction accuracy (correct prediction vs. actual word)
   - Compute surprisal as -log₂(P(word|context)) for each word

3. **Regressor Generation**
   - Create top-1 prediction regressor (binary: 1 if correct, 0 otherwise)
   - Create surprisal regressor (continuous: -log₂ probability)
   - Align regressors with EEG time series accounting for hemodynamic delay

4. **Statistical Modeling**
   - Perform linear regression: ERP amplitude ~ β₀ + β₁(top-1) + β₂(surprisal) + ε
   - Analyze coefficient significance and effect sizes
   - Examine topography and temporal dynamics of significant effects
   - Control for confounding variables (word length, frequency, etc.)

5. **Model Comparison**
   - Compare ERP prediction accuracy across different language model architectures
   - Evaluate impact of model size, training data, and architecture on neural alignment
  
   - Test whether scaling laws predict improved cognitive plausibility

6. **Validation**
   - Verify that only surprisal significantly correlates with language-processing ERPs
   - Confirm effects are stronger for open-class words with high semantic content
   - Check that prediction accuracy alone does not guarantee neural plausibility

## Verification
- Replicate the finding that surprisal correlates with N400 component (~200-500ms post-stimulus)
- Confirm top-1 prediction shows weaker or no correlation with ERP components
- Demonstrate that language model surprisal explains variance in ERP beyond control variables
- Validate that effects are modulated by word class (open vs. closed class words)

## References
- arXiv:2607.16549v1 - Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models
- Supplementary material and code (if available) from the arXiv page.

## Notes
- This skill assumes familiarity with EEG preprocessing pipelines and ERP analysis
- Requires access to EEG equipment or existing datasets with reading paradigms
- Surprisal calculation requires language models that output full probability distributions
- The approach can be extended to other cognitive tasks beyond reading comprehension