Evidence Inference Eval

---
name: evidence-inference-eval
description: This benchmark evaluates a model's ability to identify and classify clinical evidence spans within randomized controlled trial (RCT) documents. Specifically, it probes whether a given evidence span supports a significantly decreased, no significant difference, or significantly increased outcome relative to a clinical intervention. Use when the user wants to benchmark on Evidence Inference, or asks about evaluating this task. Reports macro-averaged F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.04177
  bibtex_key: deyoung2020evidenceinference2
  confidence: high
---

# evidence-inference-eval

> Evidence Inference 2.0: More Data, Better Models — DeYoung et al. (2020) (arXiv:2005.04177, 2020)

## What this evaluates

This benchmark evaluates a model's ability to identify and classify clinical evidence spans within randomized controlled trial (RCT) documents. Specifically, it probes whether a given evidence span supports a significantly decreased, no significant difference, or significantly increased outcome relative to a clinical intervention.

## Datasets

- **Evidence Inference** — total ?; splits: test (-1)

## Metrics

- `macro-averaged F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across three classes: significantly decreased, no significant difference, and significantly increased. Calculated as the unweighted mean of per-class F1 scores.

## Input / output format

**Input**: A clinical intervention/comparison/outcome (ICO) prompt and a candidate evidence span or sentence extracted from a full RCT document or abstract.

**Output**: One of three classification labels: 'significantly decreased', 'no significant difference', or 'significantly increased'.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    classes = ['sig_decreased', 'sig_no_diff', 'sig_increased']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models must be conditioned on the ICO prompt; unconditioned models drop ~27 F1 points.
- The evaluation uses a two-stage pipeline (evidence identification followed by classification); end-to-end scores are lower than oracle evidence baselines by ~0.08 F1.
- Abstract-only subsets yield nearly identical performance to full-document subsets, but oracle abstract models underperform oracle full-document models due to differences in reporting language.

## Evidence (verbatim from paper)

> Our main results are reported in Table 1. We make a few key observations. First, the gains over the prior state-of-the-art model — which was not BERT based — are substantial: 20+ absolute points in F-score, even beyond what one might expect to see shifting to large pre-trained models. To verify the impact of architecture changes, we experiment with randomly initialized and fine-tuned BERTs. We find that these perform worse than the original models in all instances and elide more detailed results. Second, conditioning on the ICO prompt is key; failing to do so results in substantial performance drops. Finally, we seem to have reached a plateau in terms of the performance of the BERT pipeline model; adding the newly collected training data does not budge performance (evaluated on the augmented test set). This suggests that to realize stronger performance here, we perhaps need a less naive architecture that better models the domain.

## Citation

```bibtex
@misc{deyoung2020evidenceinference2,
  title={Evidence Inference 2.0: More Data, Better Models},
  author={DeYoung et al. (2020)},
  year={2020},
  note={arXiv:2005.04177}
}
```

- arXiv: 2005.04177