Multimed Asr Eval

---
name: multimed-asr-eval
description: Evaluates multilingual automatic speech recognition (ASR) performance on medical domain audio across five languages. It probes the model's ability to accurately transcribe spoken medical terminology under diverse recording conditions, accents, and speaking roles. Use when the user wants to benchmark on MultiMed, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.14074
  bibtex_key: leduc2024multimed
  confidence: high
---

# multimed-asr-eval

> MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder — Khai Le-Duc et al. (2024) (arXiv:2409.14074, 2024)

## What this evaluates

Evaluates multilingual automatic speech recognition (ASR) performance on medical domain audio across five languages. It probes the model's ability to accurately transcribe spoken medical terminology under diverse recording conditions, accents, and speaking roles.

## Datasets

- **MultiMed** — total ?; splits: test (-1); repo https://github.com/leduckhai/MultiMed

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the minimum number of word edits (insertions, deletions, substitutions) required to transform the predicted transcript into the ground truth, normalized by the total number of words in the reference.
- `CER` — range: percent
  - Character Error Rate: the minimum number of character edits (insertions, deletions, substitutions) required to transform the predicted transcript into the ground truth, normalized by the total number of characters in the reference.

## Input / output format

**Input**: Audio recordings of medical speech in five languages (Vietnamese, English, German, French, Mandarin) with varying recording conditions, accents, and speaking roles.

**Output**: Text transcript of the spoken audio.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edits = levenshtein_distance(normalize(pred), normalize(ref))
        total_errors += edits
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- ASR metrics are highly sensitive to text normalization (e.g., handling of numbers, punctuation, case, and language-specific tokenization like spaces in Mandarin vs. Vietnamese).
- The paper evaluates both WER and CER but does not specify the exact normalization or alignment algorithm used, which can cause significant variance across different evaluation toolkits.

## Evidence (verbatim from paper)

> To assess the performance of the ASR models, we employed two standard evaluation metrics: Word Error Rate (WER) and Character Error Rate (CER).

## Citation

```bibtex
@misc{leduc2024multimed,
  title={MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder},
  author={Khai Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2409.14074}
}
```

- arXiv: 2409.14074