Maeb Eval

---
name: maeb-eval
description: Evaluates audio embedding models across 30 tasks spanning speech, music, environmental sounds, bioacoustics, emotion recognition, and cross-modal audio-text reasoning in over 100 languages. It probes the ability of models to generalize across acoustic domains, handle multilingual alignment, and perform both supervised and unsupervised audio understanding tasks. Use when the user wants to benchmark on MAEB, or asks about evaluating this task. Reports Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16008
  bibtex_key: [REDACTED]
  confidence: high
---

# maeb-eval

> MAEB: Massive Audio Embedding Benchmark — El Assadi et al. (2026) (arXiv:2602.16008, 2026)

## What this evaluates

Evaluates audio embedding models across 30 tasks spanning speech, music, environmental sounds, bioacoustics, emotion recognition, and cross-modal audio-text reasoning in over 100 languages. It probes the ability of models to generalize across acoustic domains, handle multilingual alignment, and perform both supervised and unsupervised audio understanding tasks.

## Datasets

- **MAEB** — total ?; splits: test (-1)

## Metrics

- `Average Score` **(primary)** — range: percent
  - Computed as the arithmetic mean of task-specific scores (accuracy for classification, similarity/retrieval scores for retrieval and reranking, and clustering metrics) across all 30 tasks. Reported as a percentage.
- `Borda Count` — range: other
  - A ranking aggregation method where models receive points based on their rank per task; the model with the highest total points ranks first overall.

## Input / output format

**Input**: Raw audio clips (and text prompts for cross-modal tasks).

**Output**: Fixed-dimensional embeddings used for downstream task evaluation (classification, retrieval, clustering, reranking).

## Scoring recipe

```python
task_scores = []
for task in MAEB_TASKS:
    preds = model.predict(task.inputs)
    gold = task.labels
    score = task.metric(preds, gold)
    task_scores.append(score)
return sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Assuming a single model dominates all acoustic domains; the benchmark reveals strong specialization where speech, music, and environmental models excel in disjoint areas.
- Cross-modal retrieval performance collapses in multilingual settings despite good audio-only performance, indicating a critical gap in cross-lingual audio-text alignment.
- Clustering tasks are universally challenging and poorly correlated with supervised task performance, revealing a fundamental gap in semantic structure of current embeddings.

## Evidence (verbatim from paper)

> Table 2 presents the top 30 models on the MAEB benchmark. The table includes both MAEB rank (over all 30 tasks) and Audio-only rank (over the 19 audio-only subset tasks) to highlight how models perform differently across task types. LCO-Embedding-Omni-7B ranks first overall by Borda count, achieving the highest average scores (52.2% overall, 50.3% cross-modal retrieval, 64.5% zero-shot) across all categories.

## Citation

```bibtex
@misc{elassadi2026maeb,
  title={MAEB: Massive Audio Embedding Benchmark},
  author={El Assadi et al. (2026)},
  year={2026},
  note={arXiv:2602.16008}
}
```

- arXiv: 2602.16008