Medbert De Med Bench Eval

---
name: medbert-de-med-bench-eval
description: Evaluates German medical language models on classification and named entity recognition tasks across radiology reports, clinical discharge notes, surgery reports, and public medical/general benchmarks. Use when the user wants to benchmark on Chest CT, Chest X-Ray, ICD-10 code classification on discharge notes, OPS code classification on discharge notes, OPS code classification on surgery reports, GermEval-18, Wrist NER, GraSCCo, GGPOnc, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.08179
  bibtex_key: bressem2023medbertde
  confidence: high
---

# medbert-de-med-bench-eval

> MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain — Bressem et al. (2023) (arXiv:2303.08179, 2023)

## What this evaluates

Evaluates German medical language models on classification and named entity recognition tasks across radiology reports, clinical discharge notes, surgery reports, and public medical/general benchmarks.

## Datasets

- **Chest CT** — total 2000; splits: train (1000), val (500), test (500)
- **Chest X-Ray** — total 2000; splits: train (1000), val (500), test (500)
- **ICD-10 code classification on discharge notes** — total 2000; splits: train (1000), val (500), test (500)
- **OPS code classification on discharge notes** — total 2000; splits: train (1000), val (500), test (500)
- **OPS code classification on surgery reports** — total 2000; splits: train (1000), val (500), test (500)
- **GermEval-18** — total ?; splits: test (-1)
- **Wrist NER** — total ?; splits: test (-1)
- **GraSCCo** — total ?; splits: test (-1)
- **GGPOnc** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Computed per class or globally depending on the task.
- `Macro F1` — range: [0, 1]
  - Unweighted mean of the F1 scores for each class, treating all classes equally regardless of support.
- `Micro F1` — range: [0, 1]
  - F1 score calculated globally by counting total true positives, false negatives, and false positives across all classes.
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive observations to the total predicted positives.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive observations to all observations in the actual class.
- `Token-level F1 (F1tok)` — range: [0, 1]
  - F1 score computed at the token level for NER tasks, aggregating predictions across all tokens in the sequence.

## Input / output format

**Input**: German medical text documents (e.g., radiology reports, discharge summaries, surgery reports, clinical guidelines, or general domain text).

**Output**: Predicted class labels for classification tasks; predicted entity tags for NER tasks.

## Scoring recipe

```python
def compute_classification_metrics(y_true, y_pred, y_prob=None):
    precision = precision_score(y_true, y_pred, average='macro')
    recall = recall_score(y_true, y_pred, average='macro')
    f1 = f1_score(y_true, y_pred, average='macro')
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr') if y_prob is not None else None
    return {'Precision': precision, 'Recall': recall, 'Macro F1': f1, 'AUROC': auc}

def compute_ner_metrics(y_true_tokens, y_pred_tokens):
    # Token-level F1/Precision/Recall computed by flattening sequences and ignoring padding
    prec = precision_score(y_true_tokens, y_pred_tokens, average='macro')
    rec = recall_score(y_true_tokens, y_pred_tokens, average='macro')
    f1 = f1_score(y_true_tokens, y_pred_tokens, average='macro')
    return {'Prec': prec, 'Rec': rec, 'F1tok': f1}
```

## Common pitfalls

- Private benchmarks (Chest CT, X-Ray, ICD-10, OPS, Wrist NER) are not publicly available, limiting independent reproduction.
- Token-level vs. document-level metrics are reported separately but the text does not specify how token-level AUROC/F1 is aggregated across sequences.
- Deduplication variants are evaluated but show negligible performance differences, making it unclear if data cleaning pipelines are standardized.

## Evidence (verbatim from paper)

> In the chest x-ray task, we found that the two best performing models were our own pre-trained BERT models. Our model trained on the corpus with duplicates removed (medBERT.dededup) achieves a slightly better performance with an average AUROC of 83.65 compared to 83.42 of the model trained on the whole corpus (medBERT.de).

## Citation

```bibtex
@misc{bressem2023medbertde,
  title={MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain},
  author={Bressem et al. (2023)},
  year={2023},
  note={arXiv:2303.08179}
}
```

- arXiv: 2303.08179