Pio Detailed Labeling Eval

---
name: pio-detailed-labeling-eval
description: Evaluates a model's ability to predict fine-grained hierarchical labels for tokens within identified P, I, and O spans. Use when the user wants to benchmark on EBM-NLP, or asks about evaluating this task. Reports F-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.04185
  bibtex_key: nye2018ebmnlp
  confidence: high
---

# pio-detailed-labeling-eval

> A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)

## What this evaluates

Evaluates a model's ability to predict fine-grained hierarchical labels for tokens within identified P, I, and O spans.

## Datasets

- **EBM-NLP** — total 5000; splits: train (4300), dev (500), test (200)

## Metrics

- `F-1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed per element type (P, I, O) and averaged across the test set.

## Input / output format

**Input**: Raw clinical trial abstract text.

**Output**: Token-level hierarchical labels from the PIO hierarchy, predicted only for tokens within identified spans.

## Scoring recipe

```python
def compute_f1(predictions, gold):
  pred_labels = [p for p in predictions if p != 'O']
  gold_labels = [g for g in gold if g != 'O']
  tp = sum(1 for p, g in zip(pred_labels, gold_labels) if p == g)
  fp = len(pred_labels) - tp
  fn = len(gold_labels) - tp
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  return 2 * p * r / (p + r) if (p + r) > 0 else 0
```

## Common pitfalls

- Initial neural models proved unfruitful, indicating high complexity and potential need for better feature engineering or pre-training.
- Labels are hierarchical; predicting incorrect sub-labels within a span counts as errors.

## Evidence (verbatim from paper)

> Beyond identifying the spans of text containing information pertinent to each of the PIO elements, we consider the task of predicting which of the detailed labels occur in each span, and where they are located. Specifically, we begin with the starting spans and predict a single label from the corresponding PIO hierarchy for each token, evaluating against the test set of 200 documents. Table 13 reports Precision, Recall, and F-1.

## Citation

```bibtex
@misc{nye2018ebmnlp,
  title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
  author={Nye et al. (2018)},
  year={2018},
  note={arXiv:1806.04185}
}
```

- arXiv: 1806.04185