Redundancy Detection Eval

---
name: redundancy-detection-eval
description: Evaluates a model's ability to determine whether pairs of P, I, or O spans in an abstract refer to the same underlying information. Use when the user wants to benchmark on EBM-NLP, or asks about evaluating this task. Reports F-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.04185
  bibtex_key: nye2018ebmnlp
  confidence: high
---

# redundancy-detection-eval

> A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature — Nye et al. (2018) (arXiv:1806.04185, 2018)

## What this evaluates

Evaluates a model's ability to determine whether pairs of P, I, or O spans in an abstract refer to the same underlying information.

## Datasets

- **EBM-NLP** — total 5000; splits: train (4300), dev (500), test (200)

## Metrics

- `F-1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: (2 * P * R) / (P + R). Computed for binary redundancy classification across all span pairs.

## Input / output format

**Input**: Pairs of P, I, or O spans extracted from a single abstract.

**Output**: Binary label (redundant or not) indicating whether the pair shares at least one instance of the same information.

## Scoring recipe

```python
def compute_f1(predictions, gold):
  tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
  fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
  fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  return 2 * p * r / (p + r) if (p + r) > 0 else 0
```

## Common pitfalls

- Task is defined over pairs of spans, not single spans, requiring a different input pipeline.
- A large majority of spans contain only a single instance of relevant information, making the task relatively easy as a baseline.

## Evidence (verbatim from paper)

> To formalize repetition, we consider every pair of starting PIO spans from each abstract, and assign binary labels that indicate whether they share at least one instance of the same information. We train a logistic regression model that operates over standard features, including bag-of-words representations and sentence-level features such as length and position in the document. Table 14 reports Precision, Recall, and F-1.

## Citation

```bibtex
@misc{nye2018ebmnlp,
  title={A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature},
  author={Nye et al. (2018)},
  year={2018},
  note={arXiv:1806.04185}
}
```

- arXiv: 1806.04185