Tamil Kannada Asr Subword Eval

---
name: tamil-kannada-asr-subword-eval
description: Evaluates automatic speech recognition (ASR) systems on agglutinative languages (Tamil and Kannada) to measure how subword dictionary learning and segmentation techniques (Morfessor, BPE, extended-BPE) reduce out-of-vocabulary rates and improve word error rates compared to baseline word-level models. Use when the user wants to benchmark on Tamil and Kannada ASR dataset, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.13331
  bibtex_key: madhavaraj2022subword
  confidence: high
---

# tamil-kannada-asr-subword-eval

> Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada — Madhavaraj et al. (2022) (arXiv:2207.13331, 2022)

## What this evaluates

Evaluates automatic speech recognition (ASR) systems on agglutinative languages (Tamil and Kannada) to measure how subword dictionary learning and segmentation techniques (Morfessor, BPE, extended-BPE) reduce out-of-vocabulary rates and improve word error rates compared to baseline word-level models.

## Datasets

- **Tamil and Kannada ASR dataset** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate (WER) measures the percentage of words incorrectly recognized in a transcript relative to the ground truth reference. Calculated as (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, and N=total words in the reference transcript.
- `OOV rate` — range: percent
  - Out-of-Vocabulary (OOV) rate is defined as the ratio of the number of words in the test data that are not present in the training corpus to the total number of words in the test data.

## Input / output format

**Input**: Audio speech recordings in Tamil or Kannada.

**Output**: Transcribed text sequence corresponding to the audio input.

## Scoring recipe

```python
def compute_wer(hypothesis, reference):
    edits = levenshtein_distance(hypothesis.split(), reference.split())
    return (edits / len(reference.split())) * 100

def compute_oov_rate(test_transcript, training_vocab):
    test_words = test_transcript.split()
    oov_count = sum(1 for w in test_words if w not in training_vocab)
    return (oov_count / len(test_words)) * 100
```

## Common pitfalls

- OOV rate is calculated against the training vocabulary, not the subword dictionary size, so a 0% OOV rate does not imply perfect ASR accuracy.
- WER improvements are highly dependent on the subword n-gram language model order (3-gram to 6-gram); comparing across different LM orders without normalization is misleading.
- ML and Viterbi segmentation techniques differ in how they estimate subword parameters (soft-weighing all paths vs. best path), which directly impacts WER and should not be conflated with dictionary creation method effects.

## Evidence (verbatim from paper)

> OOV rate is defined as the ratio of number of words in the test data which are not present in the training corpus to the total number of words in the test data. We compare the performances of subword-ASRs with the baseline ASR system in terms of OOV rate and WER and empirically justify the need for subword modeling to handle highly agglutinative languages like Tamil and Kannada.

## Citation

```bibtex
@misc{madhavaraj2022subword,
  title={Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada},
  author={Madhavaraj et al. (2022)},
  year={2022},
  note={arXiv:2207.13331}
}
```

- arXiv: 2207.13331