Umls Concept Extraction Eval

---
name: umls-concept-extraction-eval
description: Evaluates the ability of deep learning models to perform fine-grained named entity recognition for UMLS semantic types in biomedical text. It probes how well models handle class imbalance, contextual ambiguity, and domain shift between clinical notes and biomedical abstracts. Use when the user wants to benchmark on i2b2 2010, MedMentions(full), MedMentions(st21pv), or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.01274
  bibtex_key: [REDACTED]
  confidence: high
---

# umls-concept-extraction-eval

> Extracting UMLS Concepts from Medical Text Using General and Domain-Specific Deep Learning Models — Fraser et al. (2019) (arXiv:1910.01274, 2019)

## What this evaluates

Evaluates the ability of deep learning models to perform fine-grained named entity recognition for UMLS semantic types in biomedical text. It probes how well models handle class imbalance, contextual ambiguity, and domain shift between clinical notes and biomedical abstracts.

## Datasets

- **i2b2 2010** — total ?; splits: test (-1)
- **MedMentions(full)** — total ?; splits: test (-1)
- **MedMentions(st21pv)** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Standard F1-score for entity recognition: F1 = 2 * (Precision * Recall) / (Precision + Recall), computed over exact span and label matches.

## Input / output format

**Input**: Raw biomedical text (clinical notes or abstracts) containing entity spans to be identified.

**Output**: Sequence of entity spans with corresponding UMLS semantic type labels (token-level tagging implied by NER task).

## Scoring recipe

```python
def compute_f1(pred_spans, gold_spans):
    tp = len(set(pred_spans) & set(gold_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- High class ambiguity where identical text spans receive different UMLS labels depending on context.
- Severe class imbalance with many fine-grained semantic types and very few training examples per class.
- Performance drops significantly when models trained on clinical notes are evaluated on biomedical abstracts due to domain shift.

## Evidence (verbatim from paper)

> The BERT models offer a substantial improvement in F1 over the models based on Glove or ELMo embeddings for each of the three datasets.

## Citation

```bibtex
@misc{fraser2019extracting,
  title={Extracting UMLS Concepts from Medical Text Using General and Domain-Specific Deep Learning Models},
  author={Fraser et al. (2019)},
  year={2019},
  note={arXiv:1910.01274}
}
```

- arXiv: 1910.01274