Vietmed Asr Eval

---
name: vietmed-asr-eval
description: Evaluates automatic speech recognition (ASR) performance on Vietnamese medical domain audio. It measures how well models transcribe speech containing medical terminology and regional accents, assessing cross-domain transfer capabilities. Use when the user wants to benchmark on VietMed, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.05659
  bibtex_key: leduc2024vietmed
  confidence: medium
---

# vietmed-asr-eval

> VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain — Le-Duc (2024) (arXiv:2404.05659, 2024)

## What this evaluates

Evaluates automatic speech recognition (ASR) performance on Vietnamese medical domain audio. It measures how well models transcribe speech containing medical terminology and regional accents, assessing cross-domain transfer capabilities.

## Datasets

- **VietMed** — total ?; splits: dev (-1); repo https://github.com/leduckhai/MultiMed

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference transcript.

## Input / output format

**Input**: Audio recordings of Vietnamese speech in the medical domain, typically sampled at 8kHz or higher.

**Output**: Text transcript corresponding to the input audio.

## Scoring recipe

```python
def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- The best model epoch is selected based on WER on the development set, which may introduce validation set leakage if not properly separated from the final test set.
- The acoustic model uses generalized triphone states with 4501 labels, which can cause phoneme alignment issues for rare medical terms, artificially inflating WER.

## Evidence (verbatim from paper)

> We then chose the pre-training epoch to fine-tune with Framewise Cross-Entropy (fCE) loss that led to the best WERs on dev.

## Citation

```bibtex
@misc{leduc2024vietmed,
  title={VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain},
  author={Le-Duc (2024)},
  year={2024},
  note={arXiv:2404.05659}
}
```

- arXiv: 2404.05659