Faetar Benchmark Eval

---
name: faetar-benchmark-eval
description: Evaluates automatic speech recognition (ASR) models on a highly under-resourced language (Faetar/Franco-Provençal) characterized by noisy field recordings, lack of standard orthography, and inconsistent phonetic transcriptions. Use when the user wants to benchmark on Faetar Benchmark, or asks about evaluating this task. Reports PER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.08103
  bibtex_key: ong2024faetar
  confidence: high
---

# faetar-benchmark-eval

> The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language — Ong et al. (2024) (arXiv:2409.08103, 2024)

## What this evaluates

Evaluates automatic speech recognition (ASR) models on a highly under-resourced language (Faetar/Franco-Provençal) characterized by noisy field recordings, lack of standard orthography, and inconsistent phonetic transcriptions.

## Datasets

- **Faetar Benchmark** — total ?; splits: test (-1)

## Metrics

- `PER` **(primary)** — range: percent
  - Phone Error Rate calculated as the total number of alignment errors (substitutions, insertions, deletions) between the reference and hypothesis transcripts divided by the total number of phones in the reference transcript, multiplied by 100. Alignment uses the unit-cost Levenshtein algorithm.

## Input / output format

**Input**: Audio recordings of Faetar speech utterances

**Output**: Phonetic hypothesis transcript

## Scoring recipe

```python
total_errors = 0
total_ref_phones = 0
for u in test_set:
    ref = get_reference_phones(u)
    hyp = get_hypothesis_phones(u)
    errors = levenshtein_distance(ref, hyp, unit_cost=True)
    total_errors += errors
    total_ref_phones += len(ref)
per = (total_errors / total_ref_phones) * 100
```

## Common pitfalls

- Using Word Error Rate (WER) instead of PER, which is too severe due to substantial phonetic variation in the transcriptions.
- Using Character Error Rate (CER) without accounting for how spaces, affricates, and long phones are treated, which can cause minor but notable deviations from PER.

## Evidence (verbatim from paper)

> The measure of performance is phone error rate (PER) on the test set, calculated by aligning reference and hypothesis transcripts by the unit-cost Levenshtein algorithm. Given all utterances $u$ in a partition $\mathcal{U}$, the number of phones in a reference transcript $n_{u}$, and the number of errors in a model’s hypothesis transcript $e_{u}$, the PER on $\mathcal{U}$ is defined as: $\text{PER}_{\mathcal{U}}\=\frac{\sum_{u\in\mathcal{U}}e_{u}}{\sum_{u\in\mathcal{U}}n_{u}}\times 100\%.$

## Citation

```bibtex
@misc{ong2024faetar,
  title={The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language},
  author={Ong et al. (2024)},
  year={2024},
  note={arXiv:2409.08103}
}
```

- arXiv: 2409.08103