Graphpb Mos Eval

---
name: graphpb-mos-eval
description: Evaluates the naturalness and prosody quality of synthesized Chinese speech by measuring how closely the generated audio matches human-like pausing and rhythm. It probes the model's ability to capture hierarchical syntactic-semantic dependencies for prosody boundary prediction in text-to-speech systems. Use when the user wants to benchmark on Databaker dataset, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.02626
  bibtex_key: sun2020graphpb
  confidence: high
---

# graphpb-mos-eval

> GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis — Sun et al. (2020) (arXiv:2012.02626, 2020)

## What this evaluates

Evaluates the naturalness and prosody quality of synthesized Chinese speech by measuring how closely the generated audio matches human-like pausing and rhythm. It probes the model's ability to capture hierarchical syntactic-semantic dependencies for prosody boundary prediction in text-to-speech systems.

## Datasets

- **Databaker dataset** — total 10000; splits: test (100)

## Metrics

- `MOS` **(primary)** — range: [0, 5]
  - Mean Opinion Score scaled from 0 to 5 in 0.5 increments. Calculated as the average rating given by at least 10 native speakers per sentence, averaged across 100 randomly selected test sentences.

## Input / output format

**Input**: Text (graphemes/phonemes) with prosodic boundary labels (PPH/IPH edges) and corresponding acoustic features (80-dim mel-spectrograms).

**Output**: Synthesized audio waveform (generated via Griffin-Lim vocoder from mel-spectrograms) for subjective listening evaluation.

## Scoring recipe

```python
# For each of the 100 test sentences:
# 1. Generate audio using the model.
# 2. Have ≥10 native speakers listen and rate on a 0-5 scale (0.5 steps).
# 3. Compute average rating per sentence.
# 4. Average the per-sentence ratings across all 100 sentences to get final MOS.
# Report MOS ± variance.
```

## Common pitfalls

- MOS is highly subjective and depends on the specific crowd-sourcing platform and rater pool; results may not generalize across different listener demographics.
- The evaluation only uses 100 test sentences, which may not capture full dataset variability or long-form prosody issues.
- Variance is reported but not statistically tested (e.g., confidence intervals or p-values), making robustness claims qualitative.

## Evidence (verbatim from paper)

> The subjective evaluation metric chosen in this paper is Mean Opinion Score (MOS), scaling from 0 - 5 with stages increased by 0.5. The listening tests are rated by 50 native speakers on 100 randomly chosen test sentences. Each sentence is scored by at least 10 raters.

## Citation

```bibtex
@misc{sun2020graphpb,
  title={GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis},
  author={Sun et al. (2020)},
  year={2020},
  note={arXiv:2012.02626}
}
```

- arXiv: 2012.02626