Cross Lingual F5 Tts Eval

---
name: cross-lingual-f5-tts-eval
description: Evaluates the intelligibility, speaker similarity, and naturalness of synthesized speech in cross-lingual voice cloning and TTS scenarios. It also measures the accuracy of a language-agnostic speaking rate predictor for duration modeling across multiple languages. Use when the user wants to benchmark on Emilia, Seed-TTS-eval, LibriSpeech-PC test-clean, FLEURS, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.14579
  bibtex_key: liu2025crosslingualf5tts
  confidence: high
---

# cross-lingual-f5-tts-eval

> Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis — Liu et al. (2025) (arXiv:2509.14579, 2025)

## What this evaluates

Evaluates the intelligibility, speaker similarity, and naturalness of synthesized speech in cross-lingual voice cloning and TTS scenarios. It also measures the accuracy of a language-agnostic speaking rate predictor for duration modeling across multiple languages.

## Datasets

- **Emilia** — total ?; splits: train (-1)
- **Seed-TTS-eval** — total ?; splits: test-en (-1), test-zh (-1)
- **LibriSpeech-PC test-clean** — total ?; splits: test-clean (-1)
- **FLEURS** — total 473; splits: test (473)

## Metrics

- `WER` **(primary)** — range: percent
  - Measures intelligibility by comparing the automatic transcription of synthesized speech with the ground truth text using Whisper-large-V3 or Paraformer-zh.
- `SIM-o` — range: [0, 1]
  - Quantifies speaker resemblance by extracting embeddings with a WavLM-large-based verification model and computing the cosine similarity between reference and synthesized audio.
- `UTMOS` — range: other
  - Provides an automated assessment of speech naturalness using a pre-trained MOS prediction model that estimates audio quality without reference recordings.
- `MRE` — range: percent
  - Measures relative duration prediction accuracy as the average relative difference between predicted duration (linguistic units divided by predicted speaking rate) and ground truth audio duration.
- `MAE` — range: other
  - Quantifies absolute duration prediction deviation as the mean absolute difference between predicted and ground truth audio duration.

## Input / output format

**Input**: Audio prompt (3–8 seconds), target text, and reference audio (for speaker similarity evaluation).

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
# WER
pred_text = whisper_large_v3.synthesize(synthesized_audio)
wer = compute_wer(pred_text, ground_truth_text)

# SIM-o
ref_emb = wavlm_large(reference_audio)
syn_emb = wavlm_large(synthesized_audio)
sim_o = cosine_similarity(ref_emb, syn_emb)

# UTMOS
utmos_score = utmos_model.predict(synthesized_audio)

# MAE & MRE (speaking rate predictor)
pred_duration = num_linguistic_units / predicted_speaking_rate
mae = mean(abs(pred_duration - ground_truth_duration))
mre = mean(abs(pred_duration - ground_truth_duration) / ground_truth_duration)
```

## Common pitfalls

- Using Whisper-X transcriptions directly without the paper's specialized preprocessing to skip anomalous tokens (digits, symbols, other languages) causes forced alignment failures.
- Confusing the speaking rate predictor's evaluation metrics (MAE/MRE on duration) with the main TTS generation metrics (WER/SIM-o/UTMOS).
- Not adhering to the specified inference hyperparameters (Euler ODE solver, NFE=32, CFG=2.0, sway sampling=-1.0, Vocos vocoder), which significantly impact synthesis quality.

## Evidence (verbatim from paper)

> We follow the evaluation setting of F5-TTS, adopting Seed-TTS-eval and LibriSpeech-PC test-clean*[[20]]* as our test set. We also build a multilingual cross-lingual test set with 473 samples of 3-8 second audio prompts from FLEURS*[[21]]*, covering four languages (German, French, Hindi, Korean) to synthesize both English and Chinese speech. The evaluation is conducted using the following three metrics: Word Error Rate (WER) measures the intelligibility of synthesized speech by comparing its transcription with the ground truth text. We employ Whisper-large-V3*[[22]]* and Paraformer-zh*[[23]]* for automatic recognition and compute WER accordingly. Speaker Similarity (SIM-o) quantifies the resemblance between the synthesized and the original target speeches. We use WavLM-large-based*[[24]]* speaker verification model to extract speaker embeddings and compute the cosine similarity between them. UTMOS*[[25]]* provides an automated assessment of speech naturalness through a pre-trained MOS prediction model.

## Citation

```bibtex
@misc{liu2025crosslingualf5tts,
  title={Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.14579}
}
```

- arXiv: 2509.14579