Niletts Eval

---
name: niletts-eval
description: Evaluates the quality of a fine-tuned Text-to-Speech model for Egyptian Arabic dialect synthesis. It measures speech intelligibility, acoustic fidelity, and speaker similarity compared to a baseline model. Use when the user wants to benchmark on NileTTS, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.15675
  bibtex_key: [REDACTED]
  confidence: high
---

# niletts-eval

> LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models — Khamis et al. (2026) (arXiv:2602.15675, 2026)

## What this evaluates

Evaluates the quality of a fine-tuned Text-to-Speech model for Egyptian Arabic dialect synthesis. It measures speech intelligibility, acoustic fidelity, and speaker similarity compared to a baseline model.

## Datasets

- **NileTTS** — total ?; splits: test (-1); repo https://github.com/KickItLikeShika/NileTTS

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Computed by synthesizing speech from text, transcribing it with Whisper Large, and calculating the word-level edit distance against the original text. Lower values indicate higher intelligibility.
- `Character Error Rate (CER)` — range: percent
  - A finer-grained intelligibility metric computed at the character level using the same Whisper transcription and original text comparison.
- `Speaker Similarity` — range: [-1, 1]
  - Cosine similarity between ECAPA-TDNN speaker embeddings extracted from the synthesized audio and the original reference audio. Higher values indicate better voice cloning.
- `Evaluation Loss` — range: other
  - Combined text and mel-spectrogram cross-entropy loss as defined by the XTTS architecture.

## Input / output format

**Input**: Egyptian Arabic text prompts and reference speaker audio for conditioning.

**Output**: Synthesized audio waveform corresponding to the input text.

## Scoring recipe

```python
def compute_metrics(text, ref_audio, tts_model):
    syn_audio = tts_model.generate(text, ref_audio)
    transcribed = whisper_large.transcribe(syn_audio)
    wer = compute_wer(transcribed, text)
    cer = compute_cer(transcribed, text)
    ref_emb = ecapa_tdnn(ref_audio)
    syn_emb = ecapa_tdnn(syn_audio)
    spk_sim = cosine_similarity(ref_emb, syn_emb)
    return {'WER': wer, 'CER': cer, 'Spk_Sim': spk_sim}
```

## Common pitfalls

- WER and CER rely on Whisper Large for transcription, which may introduce its own biases or errors when processing Egyptian Arabic dialectal speech.
- Speaker similarity uses ECAPA-TDNN embeddings trained on standard/clean speech, potentially underestimating voice fidelity for dialectal variations.
- Checkpoint selection is automated based on metric stabilization (~34k steps), but manual listening validation was only performed on 50 randomly sampled utterances.

## Evidence (verbatim from paper)

> We use the following evaluation metrics, computed on the held-out evaluation set: Evaluation Loss: Combined text and mel-spectrogram cross-entropy loss as defined by the XTTS architecture. Word Error Rate (WER): We synthesize speech from text, transcribe it using Whisper Large Radford et al. ([2022]), and compute WER against the original text. Lower WER indicates higher intelligibility. Character Error Rate (CER): A finer-grained intelligibility metric computed at the character level. Speaker Similarity: Cosine similarity between ECAPA-TDNN Desplanques et al. ([2020]) speaker embeddings of synthesized and reference audio. Higher similarity indicates better voice cloning.

## Citation

```bibtex
@misc{khamis2026niletts,
  title={LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models},
  author={Khamis et al. (2026)},
  year={2026},
  note={arXiv:2602.15675}
}
```

- arXiv: 2602.15675