Vaani Asr Lid Eval

---
name: vaani-asr-lid-eval
description: This evaluation protocol assesses the utility of the Vaani dataset for fine-tuning automatic speech recognition (ASR) and spoken language identification (LID) models across diverse Indian languages and regions. It measures performance gains from fine-tuning on Vaani's transcribed audio and images against established benchmarks, highlighting regional dialectal variations and low-resource language capabilities. Use when the user wants to benchmark on Vaani, FLEURS, Kathbath, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28714
  bibtex_key: pulikodan2026vaani
  confidence: high
---

# vaani-asr-lid-eval

> VAANI: Capturing the language landscape for an inclusive digital India — Pulikodan et al. (2026) (arXiv:2603.28714, 2026)

## What this evaluates

This evaluation protocol assesses the utility of the Vaani dataset for fine-tuning automatic speech recognition (ASR) and spoken language identification (LID) models across diverse Indian languages and regions. It measures performance gains from fine-tuning on Vaani's transcribed audio and images against established benchmarks, highlighting regional dialectal variations and low-resource language capabilities.

## Datasets

- **Vaani** — total ?; splits: train (-1), val (-1), test (-1)
- **FLEURS** — total ?; splits: test (-1)
- **Kathbath** — total ?; splits: test (-1)

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - The percentage of words incorrectly recognized (substitutions, deletions, insertions) relative to the total number of words in the reference transcription. Lower values indicate better performance.
- `Accuracy` — range: percent
  - The percentage of correctly predicted language labels out of the total number of test instances. Per-language f1-scores are also reported as percentages.

## Input / output format

**Input**: Audio recordings of spontaneous speech (and corresponding images for multimodal tasks), with ground-truth transcriptions for ASR and language labels for LID.

**Output**: Predicted word sequences (ASR) or discrete language class labels (LID).

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    errors = 0
    for pred, ref in zip(predictions, references):
        errors += edit_distance(pred.split(), ref.split())
    return (errors / total_words) * 100

def compute_accuracy(predictions, labels):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    return (correct / len(labels)) * 100
```

## Common pitfalls

- Evaluating low-resource languages (e.g., Chakma, Bhojpuri) only on Vaani due to the absence of other public benchmarks, which limits cross-dataset comparability.
- Forcing output normalization on models like Gemma-3n-2B to remove repeated words before computing WER, which may artificially inflate scores compared to standard ASR evaluation pipelines.
- Region-specific fine-tuning results show strong geographic proximity bias, meaning performance drops significantly for distant states even within the same language, which can mislead generalization claims.

## Evidence (verbatim from paper)

> We fine-tuned the Whisper-small model (244M parameters)*Radford et al. ([2022](#bib.bib26 "Robust Speech Recognition via Large-Scale Weak Supervision"))* separately for Hindi (331 hours), Kannada (80.244 hours), Telugu (69.039 hours), and Bengali (101 hours) on a subset of the Vaani dataset, and evaluated each model independently on multiple benchmark datasets ... We normalized the output of Gemma-3n-2B when computing WER for the non-finetuned model to remove repeated words.

## Citation

```bibtex
@misc{pulikodan2026vaani,
  title={VAANI: Capturing the language landscape for an inclusive digital India},
  author={Pulikodan et al. (2026)},
  year={2026},
  note={arXiv:2603.28714}
}
```

- arXiv: 2603.28714