Mos Bench Eval

---
name: mos-bench-eval
description: This benchmark evaluates the out-of-domain generalization and robustness of subjective speech quality assessment (SSQA) models. It probes whether models trained on single or multiple datasets can accurately predict human-perceived quality scores across diverse conditions, including different languages, speech types (TTS, voice conversion, enhancement, noisy), and sampling frequencies. Use when the user wants to benchmark on MOS-Bench, or asks about evaluating this task. Reports Best score difference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.03715
  bibtex_key: huang2024mosbench
  confidence: high
---

# mos-bench-eval

> MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models — Huang et al. (2024) (arXiv:2411.03715, 2024)

## What this evaluates

This benchmark evaluates the out-of-domain generalization and robustness of subjective speech quality assessment (SSQA) models. It probes whether models trained on single or multiple datasets can accurately predict human-perceived quality scores across diverse conditions, including different languages, speech types (TTS, voice conversion, enhancement, noisy), and sampling frequencies.

## Datasets

- **MOS-Bench** — total ?; splits: test (-1); repo https://github.com/unilight/sheet

## Metrics

- `Sys MSE` — range: other
  - Mean squared error between predicted and ground truth MOS scores, averaged across systems. Used primarily for synthetic speech datasets.
- `Sys SRCC` — range: [-1, 1]
  - Spearman rank correlation coefficient between predicted and ground truth system-level average MOS scores. Used to evaluate ranking fidelity.
- `Utt MSE` — range: other
  - Mean squared error between predicted and ground truth utterance-level MOS scores. Used primarily for non-synthetic speech datasets.
- `Utt LCC` — range: [-1, 1]
  - Linear correlation coefficient (Pearson) between predicted and ground truth utterance-level MOS scores.
- `Best score difference` **(primary)** — range: other
  - MSE of a specific model minus the MSE of the best-performing model on the same test set. Averaged across all 12 test sets to assess overall generalization.
- `Best score ratio` — range: [0, 1]
  - LCC or SRCC of a specific model divided by the LCC or SRCC of the best-performing model on the same test set. Averaged across all 12 test sets.

## Input / output format

**Input**: Raw speech audio waveform, explicitly resampled to 16 kHz.

**Output**: A single continuous float representing the predicted subjective mean opinion score (MOS).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    mse = np.mean((preds - golds) ** 2)
    lcc = np.corrcoef(preds, golds)[0, 1]
    srcc = spearmanr(preds, golds).correlation
    return {'MSE': mse, 'LCC': lcc, 'SRCC': srcc}

def compute_best_score_diff_ratio(model_res, best_res):
    diff = model_res['MSE'] - best_res['MSE']
    ratio = model_res['LCC'] / best_res['LCC']
    return {'best_score_difference': diff, 'best_score_ratio': ratio}
```

## Common pitfalls

- Relying solely on system-level SRCC can mask unfaithful predictions (e.g., score skewing); high MSE is required to detect this lack of faithfulness.
- Input audio must be explicitly resampled to 16 kHz, as the underlying wav2vec 2.0 encoder only accepts 16 kHz waveforms.
- The definition of the 'best performing' model varies between single-dataset and multi-dataset training experiments, which changes the baseline for best score difference/ratio calculations.

## Evidence (verbatim from paper)

> For datasets that contain mostly synthetic speech, following the VMC series, we mainly reported two evaluation metrics: system-level mean squared error (Sys MSE) and system-level spearman rank correlation coefficient (Sys SRCC)... Here we would like to emphasize the importance of the MSE metric. MSE is a useful metric to reflect the faithfulness... best score difference := MSE of a specific model - MSE of the best performing model, best score ratio := LCC/SRCC of a specific model / LCC/SRCC of the best performing model.

## Citation

```bibtex
@misc{huang2024mosbench,
  title={MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2411.03715}
}
```

- arXiv: 2411.03715