Phoneme Level Asr Eval

---
name: phoneme-level-asr-eval
description: Evaluates automatic speech recognition performance on two low-resource, phonologically complex endangered languages (Archi and Kina Rutul) at the word, character, and phoneme levels. It specifically probes how training data frequency impacts phoneme recognition accuracy and error types. Use when the user wants to benchmark on Archi & Kina Rutul ASR, or asks about evaluating this task. Reports PER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18204
  bibtex_key: [REDACTED]
  confidence: high
---

# phoneme-level-asr-eval

> Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages — Akavarapu et al. (2026) (arXiv:2604.18204, 2026)

## What this evaluates

Evaluates automatic speech recognition performance on two low-resource, phonologically complex endangered languages (Archi and Kina Rutul) at the word, character, and phoneme levels. It specifically probes how training data frequency impacts phoneme recognition accuracy and error types.

## Datasets

- **Archi & Kina Rutul ASR** — total ?; splits: train (-1), val (-1), test (-1); HF `mahesh27/archi_rutul_asr`; repo https://github.com/mahesh-ak/north_caucasian_asr

## Metrics

- `PER` **(primary)** — range: [0, 1]
  - Normalized edit distance at the phoneme level. Precision, recall, and F1 are derived from true positives (N), substitutions (S), insertions (I), and deletions (D): pr = N/(N+S+I), re = N/(N+S+D), F1 = 2·pr·re/(pr+re).
- `WER` — range: [0, 1]
  - Normalized edit distance at the word level.
- `CER` — range: [0, 1]
  - Normalized edit distance at the character level.
- `Phoneme F1` — range: [0, 1]
  - Harmonic mean of phoneme-level precision and recall: F1 = 2·pr·re/(pr+re).

## Input / output format

**Input**: Raw audio recordings of speech in Archi or Kina Rutul.

**Output**: Transcribed text in either IPA or Cyrillic script, without translation, interpretation, or punctuation.

## Scoring recipe

```python
def compute_asr_metrics(predictions, references):
    wer = normalized_edit_distance(predictions, references, level='word')
    cer = normalized_edit_distance(predictions, references, level='char')
    per = normalized_edit_distance(predictions, references, level='phoneme')
    N, S, I, D = count_phoneme_edits(predictions, references)
    pr = N / (N + S + I)
    re = N / (N + S + D)
    f1 = 2 * pr * re / (pr + re)
    return {'WER': wer, 'CER': cer, 'PER': per, 'pr': pr, 're': re, 'F1': f1}
```

## Common pitfalls

- Confusing phoneme-level error rates with word/character rates, especially given the complex phonotactics and non-standard orthographies of these languages.
- Assuming low PER stems from model architecture limitations rather than data scarcity, as performance strongly correlates with training frequency via a sigmoid curve.
- Using incorrect CTC decoding parameters; the paper specifies a 3-gram LM with alpha=beta=0.3 and a fixed beam size of 10.

## Evidence (verbatim from paper)

> We evaluate on standard metrics for ASR — word, character and phoneme error rates, respectively WER, CER and PER which are normalized edit distances respectively at levels of words, characters and phonemes. We further store the number of edits — insertions (I), deletions (D) and substitutions (S) — along with true positives (N) for each phoneme to compute phoneme-level precision (pr), recall (re) and F1 scores:

## Citation

```bibtex
@misc{akavarapu2026hard,
  title={Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages},
  author={Akavarapu et al. (2026)},
  year={2026},
  note={arXiv:2604.18204}
}
```

- arXiv: 2604.18204