Lemas Multilingual Tts Edit Eval

---
name: lemas-multilingual-tts-edit-eval
description: This benchmark evaluates multilingual text-to-speech synthesis and text-based speech editing capabilities. It probes pronunciation stability, cross-lingual generalization, and the perceptual naturalness of localized audio edits across multiple languages. Use when the user wants to benchmark on LEMAS-Dataset, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04233
  bibtex_key: zhao2026lemas
  confidence: high
---

# lemas-multilingual-tts-edit-eval

> LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models — Zhiyuan Zhao et al. (2026) (arXiv:2601.04233, 2026)

## What this evaluates

This benchmark evaluates multilingual text-to-speech synthesis and text-based speech editing capabilities. It probes pronunciation stability, cross-lingual generalization, and the perceptual naturalness of localized audio edits across multiple languages.

## Datasets

- **LEMAS-Dataset** — total ?; splits: test (10000), edit_test (20)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate measures transcription accuracy, calculated as the percentage of substitutions, deletions, and insertions relative to the total number of words in the reference text. Lower values indicate better pronunciation stability.
- `SIM` — range: [0, 1]
  - Speaker Similarity evaluates perceptual closeness to the target speaker by computing the cosine similarity between speaker embeddings extracted from the generated audio and the reference audio using WavLM-large.
- `Subjective Preference Score` — range: percent
  - Human listeners perform A/B preference tests comparing edited audio against a baseline. Scores are normalized to a 0–100 scale where 0 indicates strong preference for the baseline and 100 indicates strong preference for the model.

## Input / output format

**Input**: For TTS: target text per language. For editing: reference audio clip, target text with word-level alignment, and a mask/placeholder indicating the region to edit.

**Output**: Synthesized audio waveform (TTS) or edited audio waveform preserving the original speaker and surrounding context (Editing).

## Scoring recipe

```python
def compute_wer(generated_audio, ref_text, lang):
    asr = Paraformer_zh if lang == 'zh' else Whisper_large_v3
    transcribed = asr.transcribe(generated_audio)
    return 100 * edit_distance(transcribed, ref_text) / len(ref_text.split())

def compute_sim(ref_audio, gen_audio):
    ref_emb = wavlm_large.encode(ref_audio)
    gen_emb = wavlm_large.encode(gen_audio)
    return cosine_similarity(ref_emb, gen_emb)

# Subjective: Average normalized 0-100 preference scores from 6 human listeners per language.
```

## Common pitfalls

- Must use language-specific ASR models for WER: Paraformer-zh for Chinese and Whisper-large-v3 for all other languages.
- Vietnamese WER results for the baseline are excluded from the average calculation due to unusually high scores across repeated measurements.
- Subjective listening tests were not conducted for TTS due to lack of native evaluators for non-Chinese/English languages.

## Evidence (verbatim from paper)

> Word error rate (WER) is evaluated using FunASR Paraformer-zh for Chinese and Whisper-large-v3 for all other languages, while speaker similarity (SIM) is computed with WavLM-large, following the Seed-TTS evaluation protocol.

## Citation

```bibtex
@misc{zhao2026lemas,
  title={LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models},
  author={Zhiyuan Zhao et al. (2026)},
  year={2026},
  note={arXiv:2601.04233}
}
```

- arXiv: 2601.04233