Wmt21 Biomedical Translation Eval

---
name: wmt21-biomedical-translation-eval
description: Evaluates Chinese-to-English machine translation performance on biomedical texts. It measures how well a neural MT system can translate domain-specific terminology and syntax while handling case, punctuation, and subword tokenization conventions. Use when the user wants to benchmark on WMT21 OK-aligned biomedical test set, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.15022
  bibtex_key: li2022summer
  confidence: high
---

# wmt21-biomedical-translation-eval

> Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task — Li et al. (2022) (arXiv:2211.15022, 2022)

## What this evaluates

Evaluates Chinese-to-English machine translation performance on biomedical texts. It measures how well a neural MT system can translate domain-specific terminology and syntax while handling case, punctuation, and subword tokenization conventions.

## Datasets

- **WMT21 OK-aligned biomedical test set** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard sentence-level BLEU score computed over n-gram precisions (typically up to 4-grams) with a brevity penalty. Scores are reported as percentages multiplied by 100.

## Input / output format

**Input**: Chinese source sentences, pre-segmented by an in-house tool, normalized for punctuation, with case markers (_UU_, _U_) prepended to words, and tokenized via BPE (32K operations).

**Output**: English target sentences, tokenized via BPE (32K operations), post-processed with Moses detokenizer.perl to restore spacing and punctuation.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # predictions and references are lists of detokenized strings
    # Standard BLEU-4 with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(
        [[ref] for ref in references],
        predictions,
        smoothing_function=nltk.translate.bleu_score.SmoothingFunction().method4
    ) * 100
```

## Common pitfalls

- Case handling requires special tokens (_UU_, _U_) before scoring; raw lowercase conversion will distort BLEU.
- Detokenization must be applied to English outputs using Moses scripts before BLEU computation to match the paper's protocol.
- BPE tokenization (32K vocab) is applied to both source and target; scoring should be done on detokenized text, not raw tokens.

## Evidence (verbatim from paper)

> The experimental results of Chinese → English on WMT21 OK-aligned biomedical test set are shown in Table 2. Compared with the baseline model (Baseline_TAG), the in-domain bilingual data (+IND-BIO) provided by WMT22 shared biomedical translation task brings a huge improvement, with 6.5 point increase in BLEU score.

## Citation

```bibtex
@misc{li2022summer,
  title={Summer: WeChat Neural Machine Translation Systems for the WMT22 Biomedical Translation Task},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2211.15022}
}
```

- arXiv: 2211.15022