N2c2 Concept Relation Eval

---
name: n2c2-concept-relation-eval
description: Evaluates clinical NLP models on extracting medical concepts and their relations from clinical notes. It probes the model's ability to handle nested/overlapped concepts and assesses cross-institutional generalization across different benchmark years. Use when the user wants to benchmark on n2c2 2018, n2c2 2022, n2c2 cross-institution (MIMIC-train/UW-test), or asks about evaluating this task. Reports strict micro-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.08262
  bibtex_key: peng2023clinicalconcept
  confidence: high
---

# n2c2-concept-relation-eval

> Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension — Peng et al. (2023) (arXiv:2303.08262, 2023)

## What this evaluates

Evaluates clinical NLP models on extracting medical concepts and their relations from clinical notes. It probes the model's ability to handle nested/overlapped concepts and assesses cross-institutional generalization across different benchmark years.

## Datasets

- **n2c2 2018** — total ?; splits: test (-1)
- **n2c2 2022** — total ?; splits: test (-1)
- **n2c2 cross-institution (MIMIC-train/UW-test)** — total ?; splits: train (-1), test (-1)

## Metrics

- `strict micro-averaged F1-score` **(primary)** — range: [0, 1]
  - Micro-averaged F1 computed over all concept and relation categories. Strict matching requires exact span boundary and type alignment between prediction and gold. Calculated using official 2018 n2c2 challenge evaluation scripts.
- `lenient micro-averaged F1-score` — range: [0, 1]
  - Micro-averaged F1 computed over all concept and relation categories. Lenient matching allows partial boundary overlap for nested/overlapped concepts.

## Input / output format

**Input**: Clinical text passages formatted as machine reading comprehension (MRC) prompts/questions.

**Output**: Extracted clinical concepts and relations, evaluated against gold annotations.

## Scoring recipe

```python
tp, fp, fn = 0, 0, 0
for pred_set, gold_set in zip(predictions, golds):
    for pred in pred_set:
        if pred in gold_set: tp += 1
        else: fp += 1
    for gold in gold_set:
        if gold not in pred_set: fn += 1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Strict vs. lenient matching criteria differ significantly for nested/overlapped concepts; using the wrong script yields misleading F1 scores.
- Metrics must be aggregated micro-averaged across all concept and relation categories, not macro-averaged or per-category.
- Official n2c2 evaluation scripts must be used; custom implementations often diverge on boundary handling.

## Evidence (verbatim from paper)

> We reported the strict and lenient micro-averaged precision, recall, and F1-score aggregated from all concept and relation categories. The official evaluation scripts provided by the 2018 n2c2 challenges were used to calculate the scores.

## Citation

```bibtex
@misc{peng2023clinicalconcept,
  title={Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2303.08262}
}
```

- arXiv: 2303.08262