Taigi Asr Eval

---
name: taigi-asr-eval
description: Evaluates the ability of automatic speech recognition (ASR) systems to transcribe Taiwanese Hokkien (Taigi) audio into text. It specifically measures how well models map Taigi phonetic patterns to Mandarin character sequences using a standardized set of public service announcement recordings. Use when the user wants to benchmark on Taigi ASR Benchmark, or asks about evaluating this task. Reports Character Error Rate (CER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.19259
  bibtex_key: lan2026breezetaigi
  confidence: high
---

# taigi-asr-eval

> Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis — Lan et al. (2026) (arXiv:2603.19259, 2026)

## What this evaluates

Evaluates the ability of automatic speech recognition (ASR) systems to transcribe Taiwanese Hokkien (Taigi) audio into text. It specifically measures how well models map Taigi phonetic patterns to Mandarin character sequences using a standardized set of public service announcement recordings.

## Datasets

- **Taigi ASR Benchmark** — total 30; splits: test (30)

## Metrics

- `Character Error Rate (CER)` **(primary)** — range: percent
  - Ratio of insertions, deletions, and substitutions to the total number of characters in the reference transcription. Computed as (I + D + S) / N, where N is the reference length. Lower values indicate better performance.

## Input / output format

**Input**: Taigi audio file (~30 seconds duration)

**Output**: Mandarin text transcription

## Scoring recipe

```python
def compute_cer(pred: str, ref: str) -> float:
    pred_norm = normalize_text(pred)  # unify numbers, punctuation, case
    ref_norm = normalize_text(ref)
    edits = levenshtein_distance(pred_norm, ref_norm)
    cer = edits / len(ref_norm) if len(ref_norm) > 0 else 0.0
    return cer * 100  # percentage
```

## Common pitfalls

- Failing to normalize outputs for numbers (Arabic vs. Han characters), punctuation, and case before computing CER.
- Interpreting absolute CER values as direct Taigi transcription accuracy, since the ground truth is in Mandarin and the Taigi-to-Mandarin mapping is not strictly one-to-one.
- Comparing systems that output Taigi orthography directly against the Mandarin ground truth without first translating the output.

## Evidence (verbatim from paper)

> We adopt CER as the primary metric, defined as the ratio of insertions, deletions, and substitutions to the total number of characters in the reference transcription. CER is more appropriate than Word Error Rate (WER) for Han character-based text because Chinese writing does not contain natural word boundaries (spaces between words).

## Citation

```bibtex
@misc{lan2026breezetaigi,
  title={Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis},
  author={Lan et al. (2026)},
  year={2026},
  note={arXiv:2603.19259}
}
```

- arXiv: 2603.19259