Indic Oov Eval

---
name: indic-oov-eval
description: This evaluation probes the out-of-vocabulary (OOV) intelligibility and perceptual quality of Indian Text-to-Speech systems. It measures how well models synthesize rare or unseen words while preserving speaker similarity and overall audio fidelity compared to ground-truth recordings. Use when the user wants to benchmark on IndicTTS, or asks about evaluating this task. Reports Intelligibility Error Rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.13435
  bibtex_key: anand2024indicoo
  confidence: high
---

# indic-oov-eval

> Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategies — Anand et al. (2024) (arXiv:2407.13435, 2024)

## What this evaluates

This evaluation probes the out-of-vocabulary (OOV) intelligibility and perceptual quality of Indian Text-to-Speech systems. It measures how well models synthesize rare or unseen words while preserving speaker similarity and overall audio fidelity compared to ground-truth recordings.

## Datasets

- **IndicTTS** — total 40; splits: train (-1)

## Metrics

- `Intelligibility Error Rate (%)` **(primary)** — range: percent
  - Percentage of words rated as unintelligible by expert listeners. Raters listen to synthesized audio alongside the source text and mark words as intelligible or not, explicitly penalizing partially intelligible words as not intelligible.
- `VISQOL` — range: other
  - Perceptual speech quality estimator using a spectro-temporal measure to compute similarity between ground-truth and synthesized speech.
- `S-SIM` — range: [-1, 1]
  - Speaker similarity score computed as the cosine similarity between embeddings extracted from ground-truth and synthesized audio samples using the Titanet model.

## Input / output format

**Input**: Synthesized audio samples paired with the corresponding source text, with the target word-of-interest highlighted.

**Output**: Binary intelligibility rating per word (intelligible / not intelligible) for human evaluation; scalar scores for VISQOL and cosine similarity for objective evaluation.

## Scoring recipe

```python
def compute_ierr(predictions, gold):
    unintelligible = sum(1 for p in predictions if p == 'not intelligible')
    return (unintelligible / len(predictions)) * 100

def compute_ssim(gt_audio, synth_audio):
    gt_emb = titanet_embed(gt_audio)
    synth_emb = titanet_embed(synth_audio)
    return cosine_similarity(gt_emb, synth_emb)
```

## Common pitfalls

- Partially intelligible words are explicitly penalized and counted as 'not intelligible', which may inflate error rates compared to lenient human transcription benchmarks.
- Raters are encouraged to discuss confusing samples, introducing potential inter-rater dependency that could reduce the independence of individual ratings.
- Objective metrics like VISQOL and S-SIM are computed on full utterances or segments, which may not isolate OOV word performance accurately.

## Evidence (verbatim from paper)

> In our evaluations, we rely on 8 such expert listeners and report the percentage of unintelligible words as the Intelligibility Error Rate (%). Next, to evaluate the perceptual quality of generated speech we rely on two objective metrics - (i) VISQOL and (ii) S-SIM or speaker similarity. To compute speaker similarity we report the cosine-similarity between the ground-truth and synthesized samples using embeddings extracted from Titanet.

## Citation

```bibtex
@misc{anand2024indicoo,
  title={Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategies},
  author={Anand et al. (2024)},
  year={2024},
  note={arXiv:2407.13435}
}
```

- arXiv: 2407.13435