Pio Span Tagging Eval

---
name: pio-span-tagging-eval
description: Evaluates a model's ability to identify text spans corresponding to Patient, Intervention, and Outcome elements within clinical trial abstracts. Use when the user wants to benchmark on EBM-NLP, or asks about evaluating this task. Reports F-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.04185
  bibtex_key: nye2018ebmnlp
  confidence: high
---

# pio-span-tagging-eval

> A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)

## What this evaluates

Evaluates a model's ability to identify text spans corresponding to Patient, Intervention, and Outcome elements within clinical trial abstracts.

## Datasets

- **EBM-NLP** — total 5000; splits: train (4300), dev (500), test (200)

## Metrics

- `F-1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed per element type (P, I, O) and averaged across the test set.

## Input / output format

**Input**: Raw clinical trial abstract text.

**Output**: Token-level BIO tags (Inside/Outside) indicating whether each token belongs to a Patient, Intervention, or Outcome span.

## Scoring recipe

```python
def compute_f1(predictions, gold):
  pred_spans = extract_spans_from_bio(predictions)
  gold_spans = extract_spans_from_bio(gold)
  tp = len(set(pred_spans) & set(gold_spans))
  fp = len(set(pred_spans) - set(gold_spans))
  fn = len(set(gold_spans) - set(pred_spans))
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  return 2 * p * r / (p + r) if (p + r) > 0 else 0
```

## Common pitfalls

- Evaluating on the development set instead of the expert-annotated test set, which compromises reliability.
- Treating the task as sentence-level classification rather than token-level span detection.

## Evidence (verbatim from paper)

> Note that we have pre-defined train, development and test sets across PIO elements for this corpus, comprising 4300, 500 and 200 abstracts, respectively. The latter set is annotated by domain experts (i.e., persons with medical training). These splits will, of course, be distributed along with the dataset to facilitate model comparisons. We treat tokens as being either Inside (I) or Outside (O) of spans. Table 12 reports Precision, Recall, and F-1 for CRF and LSTM-CRF baselines.

## Citation

```bibtex
@misc{nye2018ebmnlp,
  title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
  author={Nye et al. (2018)},
  year={2018},
  note={arXiv:1806.04185}
}
```

- arXiv: 1806.04185