Medical Entity Linking Eval

---
name: medical-entity-linking-eval
description: Evaluates a model's ability to predict semantic types for biomedical mentions and to link those mentions to standardized medical concepts. It probes how well type-based candidate filtering improves broad-coverage medical information extraction pipelines. Use when the user wants to benchmark on NCBI Disease Corpus, Bio CDR, ShARE, MedMentions, WIKIMED, PUBMEDDS, or asks about evaluating this task. Reports AUC (Area Under the Precision-Recall curve).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.00460
  bibtex_key: vashishth2020improving
  confidence: high
---

# medical-entity-linking-eval

> Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets — Vashishth et al. (2020) (arXiv:2005.00460, 2020)

## What this evaluates

Evaluates a model's ability to predict semantic types for biomedical mentions and to link those mentions to standardized medical concepts. It probes how well type-based candidate filtering improves broad-coverage medical information extraction pipelines.

## Datasets

- **NCBI Disease Corpus** — total ?; splits: test (-1)
- **Bio CDR** — total ?; splits: test (-1)
- **ShARE** — total ?; splits: test (-1)
- **MedMentions** — total ?; splits: test (-1)
- **WIKIMED** — total ?; splits: train (-1)
- **PUBMEDDS** — total ?; splits: train (-1)

## Metrics

- `AUC (Area Under the Precision-Recall curve)` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed across classification thresholds for semantic type prediction.
- `Exact Mention_id_MATCH F1` — range: [0, 1]
  - F1-score calculated on exact matches of mention spans and their linked concept IDs.
- `Partial Mention_id_MATCH F1` — range: [0, 1]
  - F1-score calculated on partial matches of mention spans and their linked concept IDs, used to isolate entity linking performance from mention detection errors.

## Input / output format

**Input**: Biomedical text snippets containing medical concept mentions, optionally with candidate concept lists for entity linking.

**Output**: Predicted semantic type labels (for type prediction) or ranked candidate concepts with CUIs (for entity linking).

## Scoring recipe

```python
def compute_f1(preds, golds, match_type='exact'):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        if match_type == 'exact' and p == g: tp += 1
        elif match_type == 'partial' and spans_overlap(p, g): tp += 1
        else: fp += 1
    fn = len(golds) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_auc(pr_scores, labels):
    return auc(pr_curve(labels, pr_scores))
```

## Common pitfalls

- Evaluating predictions for semantic types excluded from a dataset's annotation scheme (e.g., non-disease types on NCBI) leads to invalid scores; predictions must be filtered to the dataset's defined types.
- Failing to separate mention detection errors from entity linking errors; the paper isolates linking by restricting evaluation to predicted mentions that overlap with gold annotations.
- Assuming fine-grained typing is strictly necessary; the paper shows coarse-grained oracle types yield negligible performance difference compared to fine-grained ones.

## Evidence (verbatim from paper)

> Table 6 reports the results for the Exact Mention_id_MATCH and Partial Mention_id_MATCH metrics, as described in Section 5.4. ... we report the area under the precision-recall curve as our evaluation metric.

## Citation

```bibtex
@misc{vashishth2020improving,
  title={Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets},
  author={Vashishth et al. (2020)},
  year={2020},
  note={arXiv:2005.00460}
}
```

- arXiv: 2005.00460