Mos Rmbench Eval

---
name: mos-rmbench-eval
description: Evaluates the ability of speech quality reward models to correctly rank pairs of audio samples based on their Mean Opinion Score (MOS). It probes fine-grained perceptual discrimination and cross-dataset generalization in preference-based audio modeling. Use when the user wants to benchmark on BVCC, NISQA, SingMOS, SOMOS, TMHINT-QI, VMC’23, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.00743
  bibtex_key: cao2025mosrmbench
  confidence: high
---

# mos-rmbench-eval

> From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling — Cao et al. (2025) (arXiv:2510.00743, 2025)

## What this evaluates

Evaluates the ability of speech quality reward models to correctly rank pairs of audio samples based on their Mean Opinion Score (MOS). It probes fine-grained perceptual discrimination and cross-dataset generalization in preference-based audio modeling.

## Datasets

- **BVCC** — total ?; splits: test (-1)
- **NISQA** — total ?; splits: test (-1)
- **SingMOS** — total ?; splits: test (-1)
- **SOMOS** — total ?; splits: test (-1)
- **TMHINT-QI** — total ?; splits: test (-1)
- **VMC’23** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly ranked audio pairs out of the total evaluated pairs. Calculated as (correct_predictions / total_pairs) * 100.

## Input / output format

**Input**: Pairs of audio samples along with their associated MOS scores (or MOS difference information for reward shaping).

**Output**: A binary preference prediction indicating which of the two audio samples has the higher MOS, or a continuous score used to derive the ranking.

## Scoring recipe

```python
correct = 0
total = 0
for pair in dataset:
    pred = model.predict(pair.audio1, pair.audio2)
    gold = pair.audio1 if pair.mos1 > pair.mos2 else pair.audio2
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Models struggle significantly with fine-grained discrimination when the MOS difference between paired samples is small (<0.5), leading to error rates exceeding 40%.
- MOS prediction models (e.g., UTMOS) often fail to generalize across different datasets, showing high accuracy on their training domain but dropping sharply on out-of-domain benchmarks.
- Evaluating on absolute MOS scores rather than relative preferences can misrepresent model capability, as reward modeling focuses on ranking order.

## Evidence (verbatim from paper)

> As shown in the evaluation results, the Classic scalar models achieve the highest overall accuracy (80.04% with BT loss), followed by the Cloud semi-scalar models (78.82% with BT loss), while the GRMs attain slightly lower overall performance.

## Citation

```bibtex
@misc{cao2025mosrmbench,
  title={From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2510.00743}
}
```

- arXiv: 2510.00743