Neurrator Neural Semantic Narration

---
name: neurrate-neural-semantic-narration
description: NEURRATOR - Semantic narration of vision at single-cell resolution. Decodes spiking activity into natural-language descriptions using CLIP embeddings and multimodal language models.
version: 1.0
authors: [Arnau Marin-Llobet, Richard Hakim, Sara Matias, Venkatesh N. Murthy, Na Li, Demba Ba]
arxiv_id: 2606.18667
submission_date: 2026-06-17
subjects: [q-bio.NC, q-bio.QM]
keywords: [neural decoding, semantic narration, CLIP, spiking activity, single-neuron, multimodal LLM, vision, natural language, Neuropixels, mouse visual cortex]
activation_words: [neurrate, neural semantic narration, single-cell semantic decoding, CLIP neural encoding, natural language neural decoding]
---

# NEURRATOR: Semantic Narration of Vision at Single-Cell Resolution

## Overview

NEURRATOR is a groundbreaking framework that decodes spiking activity into **free-form natural-language narration** of viewed scenes at **single-neuron resolution**. This represents a paradigm shift from traditional parameterization approaches to semantic understanding of neural encoding.

## Core Innovation

### Problem Addressed
- Traditional approaches to understanding higher-order visual cortex encoding rely on:
  - Intuitive parameterization (limited effectiveness)
  - Deep-network embeddings (black boxes, uninterpretable)

### Solution: NEURRATOR Framework
1. **Learned encoder**: Maps spike trains → CLIP patch-embedding space
2. **Frozen CLIP**: No language-side training required
3. **Multimodal LLM + Sparse Autoencoder**: Generates and validates descriptions
4. **Single-neuron resolution**: Works with arbitrary neuron subsets

## Methodology

### Architecture Components

1. **Spike-to-CLIP Encoder**
   - Maps spike trains from arbitrary neuron subsets
   - Target: CLIP's patch-embedding space
   - No language-side training

2. **Frozen CLIP Model**
   - Provides semantic grounding
   - Patch embeddings as target representation

3. **Multimodal Language Model**
   - Generates natural-language descriptions
   - Works with CLIP embeddings

4. **Sparse Autoencoder (SAE)**
   - Validates generated descriptions
   - Ensures semantic fidelity

### Key Capabilities

1. **Multi-scale decoding**:
   - Thousands of neurons simultaneously
   - Single cortical regions
   - Local populations
   - Molecularly-defined cell types

2. **Quantitative analysis**:
   - Decoding fidelity vs population size
   - Regional contribution comparison
   - Cell-type functional profiling

3. **"Neurrating"**:
   - Narrate individual neuron contributions
   - Describe genetically-tagged inhibitory cell-type roles
   - Plain language functional characterization

## Experimental Application

### Dataset
- **Neuropixels recordings** of mouse visual cortex
- **Natural-movie viewing** paradigm
- Simultaneous multi-region recording

### Results
- Successful narration from:
  - Population-level recordings
  - Region-specific subsets
  - Individual neurons
  - Genetically-identified cell types

### Key Findings

1. **Decoding fidelity scaling**:
   - Population size effects
   - Regional hierarchy in semantic encoding

2. **Cell-type contributions**:
   - Inhibitory neuron functional roles
   - Molecular identity → functional probe

3. **Visual representation insights**:
   - Cell identity as functional probe
   - New biological insight units

## Technical Details

### Input Processing
- Spike trains from arbitrary neuron subsets
- Temporal encoding preservation
- Population activity patterns

### CLIP Embedding Space
- Patch-level semantic representation
- Pre-trained, frozen model
- No additional training required

### Language Generation
- Free-form natural descriptions
- Scene content narration
- Semantic validation via SAE

## Applications

### Research Applications
1. **Single-neuron characterization**: Understand individual neuron encoding
2. **Cell-type profiling**: Functional roles of molecularly-defined types
3. **Population dynamics**: Semantic interpretation of ensemble activity
4. **Regional analysis**: Compare semantic encoding across cortex

### Clinical Potential
1. **Neural prosthetics**: Semantic output generation
2. **Brain-computer interfaces**: Natural language decoding
3. **Diagnostic tools**: Functional neuron classification

## Novel Contributions

1. **Paradigm shift**: Classification target → functional probe
2. **Interpretability**: Black-box embeddings → natural language
3. **Resolution**: Population → single-neuron semantic decoding
4. **No language training**: Frozen CLIP approach

## Implementation Notes

### Requirements
- Neuropixels or similar high-density recording
- Natural scene/movie stimuli
- CLIP model access
- Multimodal LLM
- Sparse autoencoder

### Considerations
- Single-neuron noise handling
- Temporal dynamics encoding
- Semantic validation quality
- Cell-type specificity

## Related Concepts

- **Neural decoding**: Traditional vs semantic approaches
- **CLIP embeddings**: Vision-language alignment
- **Sparse autoencoders**: Interpretability tools
- **Neuropixels**: High-density recording technology
- **Natural movie viewing**: Ecological validity

## Future Directions

1. **Real-time narration**: Online semantic decoding
2. **Cross-species application**: Human visual cortex
3. **Temporal dynamics**: Dynamic scene description
4. **Memory integration**: Narrative sequence encoding
5. **Behavioral correlation**: Action semantic linking

## References

- arXiv:2606.18667
- CLIP: Radford et al., 2021
- Neuropixels: Jun et al., 2017
- Sparse autoencoders: Interpretability literature

## Conclusion

NEURRATOR transforms neural decoding from parameter fitting to semantic narration, providing a fundamentally new way to understand what individual neurons encode. The framework bridges spike trains and natural language, enabling biological insights at unprecedented resolution.