Mexpresso Mdral S2st Eval

---
name: mexpresso-mdral-s2st-eval
description: Evaluates noise-robust expressive speech-to-speech translation (S2ST) across English-Spanish and Spanish-English directions. It measures how well systems preserve naturalness and expressive style when translating speech under clean and artificially noisy conditions. Use when the user wants to benchmark on mExpresso / mDRAL, or asks about evaluating this task. Reports Naturalness MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.02733
  bibtex_key: hwang2024textless
  confidence: high
---

# mexpresso-mdral-s2st-eval

> Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation — Min-Jae Hwang et al. (2024) (arXiv:2406.02733, 2024)

## What this evaluates

Evaluates noise-robust expressive speech-to-speech translation (S2ST) across English-Spanish and Spanish-English directions. It measures how well systems preserve naturalness and expressive style when translating speech under clean and artificially noisy conditions.

## Datasets

- **mExpresso / mDRAL** — total ?; splits: dev (-1), test (-1)

## Metrics

- `Naturalness MOS` **(primary)** — range: other
  - Mean Opinion Score from human listeners rating the naturalness of synthesized speech on a typical 1-5 scale. Reported as mean ± 95% confidence interval.
- `S-MOS` — range: other
  - Style/Expressivity Mean Opinion Score from human listeners rating how well the synthesized speech preserves the source speaker's vocal style, emotion, and tone. Reported as mean ± 95% confidence interval.

## Input / output format

**Input**: Source speech audio (clean or noisy) in English or Spanish.

**Output**: Target language speech waveform (24 kHz) synthesized from XLS-R 10K units via DINO-PRETSSEL and HiFi-GAN vocoder.

## Scoring recipe

```python
def compute_metrics(predictions, gold, conditions):
    # predictions: list of generated audio waveforms
    # gold: list of reference utterances
    # conditions: list of 'clean' or 'noisy' labels
    ratings = []
    for pred, ref, cond in zip(predictions, gold, conditions):
        nat_score = human_rate_naturalness(pred)
        style_score = human_rate_style(pred, ref)
        ratings.append({'naturalness': nat_score, 'style': style_score, 'condition': cond})
    
    clean_nat = [r['naturalness'] for r in ratings if r['condition'] == 'clean']
    noisy_nat = [r['naturalness'] for r in ratings if r['condition'] == 'noisy']
    clean_style = [r['style'] for r in ratings if r['condition'] == 'clean']
    noisy_style = [r['style'] for r in ratings if r['condition'] == 'noisy']
    
    return {
        'clean_naturalness_mos': mean(clean_nat),
        'noisy_naturalness_mos': mean(noisy_nat),
        'clean_s_mos': mean(clean_style),
        'noisy_s_mos': mean(noisy_style)
    }
```

## Common pitfalls

- Confusing 'Clean' vs 'Noisy' source conditions, as results are reported separately for each condition rather than averaged.
- Ignoring the 95% confidence intervals when claiming statistical superiority between systems.
- Assuming evaluation noise matches training SNR (6dB-40dB), as test noise levels are not explicitly detailed in the protocol.

## Evidence (verbatim from paper)

> Naturalness MOS↑ | S-MOS↑ | ... Table 2: Subjective evaluation results for various expressive S2ST systems with a 95% confidence interval. The highest scores are in bold typeface. "Clean" and "Noisy" denote that the source speech of S2ST system was clean and noisy, respectively.

## Citation

```bibtex
@misc{hwang2024textless,
  title={Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation},
  author={Min-Jae Hwang et al. (2024)},
  year={2024},
  note={arXiv:2406.02733}
}
```

- arXiv: 2406.02733