Mseb Eval

---
name: mseb-eval
description: Evaluates audio embedding models and ASR pipelines across eight core auditory tasks to measure real-world generalization, compression robustness, and the performance gap between direct audio processing and text-based oracles. It quantifies how well embeddings capture semantic and acoustic information without task-specific fine-tuning. Use when the user wants to benchmark on SVQ (Simple Voice Questions), Speech-MASSIVE, FSD50K, BirdSet, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.07143
  bibtex_key: heigold2026mseb
  confidence: high
---

# mseb-eval

> Massive Sound Embedding Benchmark (MSEB) — Heigold et al. (2026) (arXiv:2602.07143, 2026)

## What this evaluates

Evaluates audio embedding models and ASR pipelines across eight core auditory tasks to measure real-world generalization, compression robustness, and the performance gap between direct audio processing and text-based oracles. It quantifies how well embeddings capture semantic and acoustic information without task-specific fine-tuning.

## Datasets

- **SVQ (Simple Voice Questions)** — total ?; splits: test (-1)
- **Speech-MASSIVE** — total ?; splits: test (-1)
- **FSD50K** — total ?; splits: test (-1)
- **BirdSet** — total ?; splits: test (-1)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of the reciprocal of the rank of the first correct item in a ranked list of candidates.
- `mAP` — range: [0, 1]
  - mean Average Precision: the area under the precision-recall curve, averaging precision across all recall levels for each query.
- `F1` — range: [0, 1]
  - F1 score: the harmonic mean of precision and recall, balancing false positives and false negatives.
- `WER` — range: percent
  - Word Error Rate: the percentage of words incorrectly transcribed relative to the ground truth, calculated as (substitutions + deletions + insertions) / total words.
- `NDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain: measures ranking quality by discounting the relevance of items based on their position in the list.
- `V-measure` — range: [0, 1]
  - V-measure: the harmonic mean of homogeneity and completeness, evaluating clustering quality against ground-truth labels.
- `FAD` — range: other
  - Fréchet Audio Distance: computes the Fréchet distance between the multivariate Gaussian distributions of features extracted from real and reconstructed audio.

## Input / output format

**Input**: Audio recordings (speech, environmental, or bioacoustic) or ground-truth text transcripts, paired with queries, candidate hypotheses, class labels, or target audio signals.

**Output**: Embedding vectors, predicted class labels, ranked candidate lists, transcribed text, or reconstructed audio waveforms.

## Scoring recipe

```python
def compute_metric(task, predictions, gold):
    if task == 'retrieval':
        return mean([1.0 / rank for rank in predictions])  # MRR
    elif task in ('reranking', 'classification'):
        return average_precision(gold, predictions)  # mAP
    elif task == 'reasoning':
        return f1_score(gold, predictions)  # F1
    elif task == 'transcription':
        return wer(gold, predictions)  # WER
    elif task == 'segmentation':
        return ndcg_at_k(gold, predictions)  # NDCG
    elif task == 'clustering':
        return v_measure_score(gold, predictions)  # V-measure
    elif task == 'reconstruction':
        return frechet_audio_distance(gold, predictions)  # FAD
```

## Common pitfalls

- Assuming ASR quality (WER) perfectly correlates with downstream task performance, as retrieval and reasoning depend disproportionately on semantically salient terms rather than literal word accuracy.
- Treating unbounded metrics like FAD or WER as having a fixed upper bound; the benchmark explicitly notes they are open-ended and visualized with dashed lines.
- Ignoring locale/domain variability; aggregate scores mask extreme performance drops for low-resource languages or noisy conditions, which the paper highlights as a key finding.

## Evidence (verbatim from paper)

> Retrieval was performed via dot product between the query embedding and document embeddings, with quality measured by Mean Reciprocal Rank (MRR).

## Citation

```bibtex
@misc{heigold2026mseb,
  title={Massive Sound Embedding Benchmark (MSEB)},
  author={Heigold et al. (2026)},
  year={2026},
  note={arXiv:2602.07143}
}
```

- arXiv: 2602.07143