Clinical Text Robustness Eval

---
name: clinical-text-robustness-eval
description: Evaluates the robustness of clinical NLP models against real-world input noise across four standard tasks. It probes whether models maintain performance when text contains character- or word-level perturbations (e.g., misspellings, deletions, negations) that remain human-readable. Use when the user wants to benchmark on i2b2, MedSTS, MedNLI, or asks about evaluating this task. Reports evaluation scores (accuracy/F1).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.12242
  bibtex_key: moradi2021clinicaltextperturbation
  confidence: high
---

# clinical-text-robustness-eval

> Deep learning models are not robust against noise in clinical text — Moradi et al. (2021) (arXiv:2108.12242, 2021)

## What this evaluates

Evaluates the robustness of clinical NLP models against real-world input noise across four standard tasks. It probes whether models maintain performance when text contains character- or word-level perturbations (e.g., misspellings, deletions, negations) that remain human-readable.

## Datasets

- **i2b2** — total ?; splits: test (-1)
- **MedSTS** — total ?; splits: test (-1)
- **MedNLI** — total ?; splits: test (-1)

## Metrics

- `evaluation scores (accuracy/F1)` **(primary)** — range: percent
  - Task-specific accuracy or F1 score calculated as (correct predictions / total predictions) * 100 or standard F1 formula. Reported as percentage.

## Input / output format

**Input**: Clinical text sentences or documents, optionally perturbed with character-level or word-level noise according to a specified Perturbation Per Sample (PPS) parameter.

**Output**: Task-specific predictions: NER entity tags, relation extraction labels, textual inference labels, or semantic similarity scores.

## Scoring recipe

```python
original_score = compute_metric(model.predict(original_text), gold)
perturbed_score = compute_metric(model.predict(perturbed_text), gold)
drop = original_score - perturbed_score
return perturbed_score, drop
```

## Common pitfalls

- Only PPS=1 results are reported in the main table, though PPS up to 4 is tested.
- Human readability does not guarantee model robustness; models fail on noise that humans find understandable.
- Task sensitivity varies significantly: semantic similarity degrades more than relation extraction under identical noise.

## Evidence (verbatim from paper)

> Table 3 also presents the evaluation scores obtained by the neural language models on the perturbed samples from the clinical NLP datasets. In the experiments, a parameter named Perturbation Per Sample (PPS) controlled the maximum number of perturbations per sample. For brevity reasons, only the results for PPS=1 are reported in Table 3.

## Citation

```bibtex
@misc{moradi2021clinicaltextperturbation,
  title={Deep learning models are not robust against noise in clinical text},
  author={Moradi et al. (2021)},
  year={2021},
  note={arXiv:2108.12242}
}
```

- arXiv: 2108.12242