Mecat Eval

---
name: mecat-eval
description: This benchmark evaluates fine-grained audio understanding by testing models on generating detailed, multi-perspective captions and answering probing questions across diverse acoustic domains. It specifically probes a model's ability to distinguish between speech, music, and sound events, reason about acoustic scenes, and assess technical audio quality without relying on generic descriptions. Use when the user wants to benchmark on MECAT, or asks about evaluating this task. Reports DATE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.23511
  bibtex_key: niu2025mecat
  confidence: high
---

# mecat-eval

> MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks — Niu et al. (2025) (arXiv:2507.23511, 2025)

## What this evaluates

This benchmark evaluates fine-grained audio understanding by testing models on generating detailed, multi-perspective captions and answering probing questions across diverse acoustic domains. It specifically probes a model's ability to distinguish between speech, music, and sound events, reason about acoustic scenes, and assess technical audio quality without relying on generic descriptions.

## Datasets

- **MECAT** — total 20000; splits: test (20000); repo https://github.com/xiaomi-research/mecat

## Metrics

- `DATE` **(primary)** — range: [0, 1]
  - DATE combines weighted semantic similarity (with penalties for generic terms) and cross-sample discriminability. The overall evaluation score is computed as a weighted average of caption and QA scores: Score_Cap = 0.4*S_Systemic + 0.4*S_Content-Specific + 0.2*S_Content-Unrelated, where S_Systemic = 0.8*S_Long + 0.2*S_Short and S_Content-Specific = 0.6*S_Speech + 0.3*S_Music + 0.1*S_Sound. Score_QA is the unweighted mean of six cognitive sub-categories (DP, SC, QAS, ER, IJ, AC).

## Input / output format

**Input**: Audio clip (maximum 10 seconds duration) paired with a specific question (for QA task) or provided as-is (for captioning task).

**Output**: Text string containing a caption (short, long, domain-specific, or content-unrelated) or a direct answer to the posed question.

## Scoring recipe

```python
S_long = DATE(pred, refs_long)
S_short = DATE(pred, refs_short)
S_speech = DATE(pred, refs_speech)
S_music = DATE(pred, refs_music)
S_sound = DATE(pred, refs_sound)
S_unrelated = DATE(pred, refs_unrelated)
S_DP = DATE(pred, refs_DP)
S_SC = DATE(pred, refs_SC)
S_QAS = DATE(pred, refs_QAS)
S_ER = DATE(pred, refs_ER)
S_IJ = DATE(pred, refs_IJ)
S_AC = DATE(pred, refs_AC)
S_systemic = 0.8 * S_long + 0.2 * S_short
S_content_specific = 0.6 * S_speech + 0.3 * S_music + 0.1 * S_sound
Score_Cap = 0.4 * S_systemic + 0.4 * S_content_specific + 0.2 * S_unrelated
Score_QA = (S_DP + S_SC + S_QAS + S_ER + S_IJ + S_AC) / 6
return Score_Cap, Score_QA
```

## Common pitfalls

- Models often generate generic captions that score poorly on DATE due to the explicit penalty for non-specific terms.
- Evaluators must ensure domain-specific captions explicitly state when a domain (e.g., silence or music) is absent, as the benchmark requires negative assertions.
- The weighted aggregation for captioning heavily prioritizes systemic and content-specific captions (0.8 weight each), so neglecting short captions or content-unrelated acoustic quality will disproportionately lower the final score.

## Evidence (verbatim from paper)

> It addresses the gap in current benchmarks by introducing DATE, a novel evaluation metric that combines weighted semantic similarity (penalizing generic terms) with cross-sample discriminability to robustly distinguish between superficial and contextually rich model outputs, enabling more accurate assessment of perceptual fidelity in audio understanding.

## Citation

```bibtex
@misc{niu2025mecat,
  title={MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks},
  author={Niu et al. (2025)},
  year={2025},
  note={arXiv:2507.23511}
}
```

- arXiv: 2507.23511