Crossvoice S2st Eval

---
name: crossvoice-s2st-eval
description: Evaluates cross-lingual speech-to-speech translation (S2ST) systems on translation accuracy and prosody preservation. It measures how well a cascade-based S2ST pipeline preserves speaker identity and naturalness while translating speech across different language pairs. Use when the user wants to benchmark on CVSS-T, Indic-TTS, Fisher, MuST-C, VoxPopuli, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.00021
  bibtex_key: hira2024crossvoice
  confidence: high
---

# crossvoice-s2st-eval

> CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning — Hira et al. (2024) (arXiv:2406.00021, 2024)

## What this evaluates

Evaluates cross-lingual speech-to-speech translation (S2ST) systems on translation accuracy and prosody preservation. It measures how well a cascade-based S2ST pipeline preserves speaker identity and naturalness while translating speech across different language pairs.

## Datasets

- **CVSS-T** — total ?; splits: test (-1)
- **Indic-TTS** — total ?; splits: test (-1)
- **Fisher** — total ?; splits: test (-1)
- **MuST-C** — total ?; splits: test (-1)
- **VoxPopuli** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Computed by first transcribing the generated speech using Whisper (temperature=1, greedy decoding), then calculating the BLEU score between the generated transcript and the ground-truth reference transcript.
- `MOS-c` — range: [0, 5]
  - Mean Opinion Score measuring subjective perceived quality/naturalness of the synthesized speech output.

## Input / output format

**Input**: Source speech audio in a target language (e.g., Spanish, German, Italian, Hindi, French).

**Output**: Translated speech audio in the source language (e.g., English).

## Scoring recipe

```python
# Transcribe generated speech for BLEU
gen_text = whisper_transcribe(generated_audio, temperature=1, decoding="greedy")
bleu = compute_bleu(reference_text, gen_text)

# Compute MOS-c
mos_c = mean([human_rating(audio) for audio in generated_audio])
```

## Common pitfalls

- BLEU is computed on Whisper-transcribed audio rather than direct text, so ASR errors directly impact the translation metric.
- SOTA BLEU scores are taken from original papers (BLEU-r) instead of being re-evaluated under identical conditions, risking unfair comparison.
- MOS scores are subjective human ratings; the paper reports mean ± std, so statistical variance must be considered when comparing systems.

## Evidence (verbatim from paper)

> For calculating the BLEU scores, we employed Whisper (using the temperature setting of one and greedy decoding) for generating transcripts of the speech generated using CrossVoice and SOTA methods. MOS-c score is almost the same as MOS-h (i.e., the GT) and also beats MOS-v scores of the vanilla TTS considerably, by almost 40% on each task.

## Citation

```bibtex
@misc{hira2024crossvoice,
  title={CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning},
  author={Hira et al. (2024)},
  year={2024},
  note={arXiv:2406.00021}
}
```

- arXiv: 2406.00021