Multimed St Eval

---
name: multimed-st-eval
description: Evaluates the capability of speech translation models to accurately convert medical speech across five languages (English, Vietnamese, German, French, Mandarin Chinese) into text. It probes both end-to-end and cascaded architectures, as well as the impact of multilingual vs. bilingual training and code-switching handling in a specialized medical domain. Use when the user wants to benchmark on MultiMed-ST, or asks about evaluating this task. Reports BLEU, BERTScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.03546
  bibtex_key: leduc2025multimedst
  confidence: high
---

# multimed-st-eval

> MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation — Le-Duc et al. (2025) (arXiv:2504.03546, 2025)

## What this evaluates

Evaluates the capability of speech translation models to accurately convert medical speech across five languages (English, Vietnamese, German, French, Mandarin Chinese) into text. It probes both end-to-end and cascaded architectures, as well as the impact of multilingual vs. bilingual training and code-switching handling in a specialized medical domain.

## Datasets

- **MultiMed-ST** — total 290000; splits: test (-1); repo https://github.com/leduckhai/MultiMed-ST

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric computing the geometric mean of modified precisions up to n=4, adjusted with a brevity penalty to penalize overly short translations. Values in the paper are scaled by 100.
- `BERTScore` **(primary)** — range: [0, 1]
  - Embedding-based metric that computes the F1 score between reference and hypothesis using contextual embeddings (e.g., BERT), matching tokens via cosine similarity.

## Input / output format

**Input**: Audio recording of medical speech in a source language (en, vi, fr, de, or zh).

**Output**: Transcribed and translated text in the target language.

## Scoring recipe

```python
def compute_bleu(hypotheses, references):
    # Standard sentence-level BLEU with n=4 and brevity penalty, scaled to 100
    return 100 * nltk.translate.bleu_score.corpus_bleu(references, hypotheses)

def compute_bertscore(hypotheses, references):
    # BERTScore F1 using contextual embeddings
    scores = bertscore.score(hypotheses, references, lang='auto')
    return scores.f1.mean()
```

## Common pitfalls

- Relying solely on n-gram metrics like BLEU may misrepresent semantic quality in medical translation; embedding-based metrics like BERTScore are recommended as complementary.
- Cascaded systems (ASR→MT) accumulate ASR errors that directly degrade translation quality, which end-to-end models may mitigate but often lack domain-specific tuning.
- Code-switching and low-resource language pairs (e.g., vi-de, zh-fr) show significantly lower performance, requiring careful data balancing or multi-task training.

## Evidence (verbatim from paper)

> Automatic MT metrics: To evaluate MT quality, two standard categories of evaluation metrics were utilized: n-gram overlap metrics (e.g., BLEU (Papineni et al. (2002)), TER (Snover et al. (2006)), METEOR (Banerjee and Lavie (2005)), ChrF (Popović (2015)), ROUGE (Lin (2004))) and embedding-based metrics (e.g., BERTScore (Zhang et al.)).

## Citation

```bibtex
@misc{leduc2025multimedst,
  title={MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation},
  author={Le-Duc et al. (2025)},
  year={2025},
  note={arXiv:2504.03546}
}
```

- arXiv: 2504.03546