Seamlessm4t Human Eval

---
name: seamlessm4t-human-eval
description: Probes the semantic preservation and audio naturalness of speech-to-text and speech-to-speech translation systems across 24+ languages. Uses human annotators to score translations on a 1-5 scale for meaning similarity (XSTS) and speech quality/naturalness (MOS). Use when the user wants to benchmark on FLEURS test partition, or asks about evaluating this task. Reports XSTS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.11596
  bibtex_key: seamlesscommunication2023seamlessm4t
  confidence: high
---

# seamlessm4t-human-eval

> SeamlessM4T: Massively Multilingual & Multimodal Machine Translation — Seamless Communication et al. (2023) (arXiv:2308.11596, 2023)

## What this evaluates

Probes the semantic preservation and audio naturalness of speech-to-text and speech-to-speech translation systems across 24+ languages. Uses human annotators to score translations on a 1-5 scale for meaning similarity (XSTS) and speech quality/naturalness (MOS).

## Datasets

- **FLEURS test partition** — total ?; splits: test (-1); HF `facebook/fleurs`

## Metrics

- `XSTS` **(primary)** — range: [1, 5]
  - Semantic similarity score on a 1-5 Likert scale. Annotators evaluate source-target pairs; median of 3 scores is taken, with additional annotators if disagreement ≥2. Scores are calibrated using a cross-lingual calibration set.
- `MOS` — range: [1, 5]
  - Mean Opinion Score on a 1-5 Likert scale assessing clarity, sound quality, and naturalness. Median of annotator scores is collected per item. No calibration or tie-breaking annotators are used.

## Input / output format

**Input**: Source audio (or text) and target audio (or text) pairs from the FLEURS test set, filtered to sentences with recordings in both languages. Includes cross-lingual calibration items.

**Output**: A single integer score from 1 to 5 per item, representing semantic similarity (XSTS) or audio quality/naturalness (MOS).

## Scoring recipe

```python
def aggregate_scores(raw_annotator_scores):
    # raw_annotator_scores: list of lists of 1-5 scores per item
    medians = [median(scores) for scores in raw_annotator_scores]
    # Calibrate using cross-lingual calibration set bias
    calibrated = [m - calibration_bias for m in medians]
    return mean(calibrated)
```

## Common pitfalls

- Annotators must be explicitly instructed to ignore non-speech tags (e.g., <laugh>) and pauses/noises during XSTS scoring, otherwise scores drop artificially.
- MOS evaluations are not calibrated and only cover a limited subset of language directions into English, so they cannot be directly compared to full-direction XSTS results.
- Using mean instead of median for annotator scores reduces robustness to outlier annotators, contrary to the paper's protocol.

## Evidence (verbatim from paper)

> For the S2ST task, we evaluate using two protocols: XSTS for translation quality, and MOS to assess naturalness. XSTS [Licht et al., 2022] evaluates translation quality in terms of semantic meaning preservation, and has previously been used to evaluate the NLLB models [NLLB Team et al., 2022]. While XSTS was originally designed to evaluate text, the protocol is effectively modality agnostic, and we required only small adaptations in order to support S2ST and S2TT tasks.

## Citation

```bibtex
@misc{seamlesscommunication2023seamlessm4t,
  title={SeamlessM4T: Massively Multilingual & Multimodal Machine Translation},
  author={Seamless Communication et al. (2023)},
  year={2023},
  note={arXiv:2308.11596}
}
```

- arXiv: 2308.11596