Voice Of India Eval

---
name: voice-of-india-eval
description: Evaluates automatic speech recognition (ASR) systems on real-world, unscripted telephonic conversations across 15 Indian languages. It probes geographic, demographic, and audio quality disparities in model performance, particularly focusing on code-mixed speech and natural orthographic variations. Use when the user wants to benchmark on Voice of India, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19151
  bibtex_key: [REDACTED]
  confidence: high
---

# voice-of-india-eval

> Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Bhogale et al. (2026) (arXiv:2604.19151, 2026)

## What this evaluates

Evaluates automatic speech recognition (ASR) systems on real-world, unscripted telephonic conversations across 15 Indian languages. It probes geographic, demographic, and audio quality disparities in model performance, particularly focusing on code-mixed speech and natural orthographic variations.

## Datasets

- **Voice of India** — total 36691; splits: test (-1)

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Standard WER = (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=reference words. This benchmark computes WER against a lattice of multiple valid transcripts to account for orthographic variation, code-mixing, and disfluencies.

## Input / output format

**Input**: Audio recordings of spontaneous telephonic conversations, segmented into utterances via WebRTC VAD.

**Output**: Text transcript of the spoken audio.

## Scoring recipe

```python
def compute_lattice_wer(predictions, lattices):
    total_wer = 0.0
    for pred, lattice in zip(predictions, lattices):
        # Find reference path in lattice that minimizes edit distance to pred
        best_ref = min(lattice.paths, key=lambda r: edit_distance(pred, r))
        total_wer += wer_score(pred, best_ref)
    return (total_wer / len(predictions)) * 100
```

## Common pitfalls

- Standard single-reference WER unfairly penalizes valid code-mixed words and natural spelling variations; evaluators must use the provided lattice/multi-reference format.
- Audio quality and demographic stratification cause significant performance variance by district and device; reporting only aggregate WER masks critical real-world disparities.
- Disfluencies and half-words are optional in the lattice; models that rigidly enforce or ignore these will be mis-scored if evaluated against a flat reference.

## Evidence (verbatim from paper)

> It uses multiple valid transcripts to mitigate WER penalties for natural spelling variation and code-mixed speech, and conducts granular geographic, demographic, and audio quality analysis revealing significant performance disparities at the district level—highlighting real-world limitations of current ASR systems in India.

## Citation

```bibtex
@misc{bhogale2026voiceofindia,
  title={Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India},
  author={Bhogale et al. (2026)},
  year={2026},
  note={arXiv:2604.19151}
}
```

- arXiv: 2604.19151