Habibi Tts Eval

---
name: habibi-tts-eval
description: Evaluates zero-shot and unified-dialectal text-to-speech synthesis across multiple Arabic dialects. It measures transcription accuracy, speaker similarity, and audio naturalness to assess how well a model preserves dialectal features and voice identity without dialect-specific fine-tuning. Use when the user wants to benchmark on Habibi Benchmark, or asks about evaluating this task. Reports WER-O.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.13802
  bibtex_key: chen2026habibi
  confidence: high
---

# habibi-tts-eval

> Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis — Chen et al. (2026) (arXiv:2601.13802, 2026)

## What this evaluates

Evaluates zero-shot and unified-dialectal text-to-speech synthesis across multiple Arabic dialects. It measures transcription accuracy, speaker similarity, and audio naturalness to assess how well a model preserves dialectal features and voice identity without dialect-specific fine-tuning.

## Datasets

- **Habibi Benchmark** — total ?; splits: test (-1)

## Metrics

- `WER-O` **(primary)** — range: percent
  - Word Error Rate computed by the Omnilingual-ASR-LLM-7B model on generated speech compared to ground truth text. Lower is better.
- `WER-S` — range: percent
  - Word Error Rate computed by dialect-specific ASR models on generated speech compared to ground truth text. Lower is better.
- `SIM` — range: [0, 1]
  - Speaker similarity score computed using the WavLM speaker verification model between the generated audio and the reference audio. Higher is better.
- `UTMOS` — range: other
  - Naturalness score predicted by the UTMOS model on the generated audio. Higher is better.

## Input / output format

**Input**: Arabic text (dialect-specific) and a reference audio clip (for zero-shot/in-context voice cloning).

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
# WER-O/S: Transcribe generated audio with ASR, compare to GT text
asr = load_asr("Omnilingual-ASR-LLM-7B" or dialect_specific)
pred = asr.transcribe(generated_audio)
wer = compute_wer(pred, ground_truth_text)

# SIM: Speaker similarity via WavLM
wavlm = load_wavlm()
sim = wavlm.compute_similarity(generated_audio, reference_audio)

# UTMOS: Naturalness score
utmos = load_utmos()
utmos_score = utmos.predict(generated_audio)
```

## Common pitfalls

- Multilingual ASR models may exhibit cross-dialect recognition bias, incorrectly 'correcting' dialectal speech, which is why the authors report both WER-O and WER-S.
- Ground truth samples from ASR corpora often contain higher noise levels than TTS outputs, which can artificially inflate WER scores for the reference speech.
- Commercial baseline comparison requires careful selection and LLM-validated reference audio to ensure authentic dialect characteristics.

## Evidence (verbatim from paper)

> We measure three conventional metrics: word error rate (WER) using ASR models, speaker similarity (SIM) leveraging the speaker verification model WavLM *Chen et al. ([2022])*, and naturalness with UTMOS *Saeki et al. ([2022])*. Notably, we report two sets of WER scores: (1) WER-O, evaluated with Omnilingual-ASR-LLM-7B model *Omnilingual et al. ([2025])* (v1 with a fixed batch size of 64); and (2) WER-S, derived from dialect-specific ASR models, most of which trained following VietASR *Zhuo et al. ([2025])* except for EGY and MAR (two XLSR fine-tuned models on Hugging Face666[IbrahimAmin/egyptian-arabic-wav2vec2-xlsr-53](https://huggingface.co/IbrahimAmin/egyptian-arabic-wav2vec2-xlsr-53 ""),777[boumehdi/wav2vec2-large-xlsr-moroccan-darija](https://huggingface.co/boumehdi/wav2vec2-large-xlsr-moroccan-darija "") are employed). The rationale for introducing both is to enable a more reliable conclusion, given that a multilingual model risks cross-dialect recognition bias (e.g., incorrectly “correcting” speech that lacks dialect features); conversely, specialized models often suffer from poor generalization and noise resistance.

## Citation

```bibtex
@misc{chen2026habibi,
  title={Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.13802}
}
```

- arXiv: 2601.13802