Pubmedqa Eval

---
name: pubmedqa-eval
description: This benchmark evaluates a model's ability to perform biomedical research question answering by reasoning over structured scientific abstracts. It requires models to infer yes/no/maybe answers to questions derived from paper titles using only the non-conclusion sections of the abstract, without access to the final conclusion. Use when the user wants to benchmark on PubMedQA (PQA-L), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.06146
  bibtex_key: jin2019pubmedqa
  confidence: high
---

# pubmedqa-eval

> PubMedQA: A Dataset for Biomedical Research Question Answering — Jin et al. (2019) (arXiv:1909.06146, 2019)

## What this evaluates

This benchmark evaluates a model's ability to perform biomedical research question answering by reasoning over structured scientific abstracts. It requires models to infer yes/no/maybe answers to questions derived from paper titles using only the non-conclusion sections of the abstract, without access to the final conclusion.

## Datasets

- **PubMedQA (PQA-L)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels (yes, no, or maybe) out of the total number of instances.
- `macro-F1` — range: percent
  - Unweighted mean of the F1 scores calculated independently for each of the three classes (yes, no, maybe).

## Input / output format

**Input**: A research question (derived from a title) and the context sections of a biomedical abstract (excluding the conclusion).

**Output**: A single categorical label: 'yes', 'no', or 'maybe'.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = ['yes', 'no', 'maybe']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(classes)
    return accuracy * 100, macro_f1 * 100
```

## Common pitfalls

- The main evaluation uses a 'reasoning-required' setting where models must predict without seeing the abstract's conclusion; evaluating in a 'reasoning-free' setting (where the conclusion is provided) yields artificially high scores.
- The pre-training subset (PQA-A) is highly imbalanced, causing a trivial majority baseline to achieve ~92% accuracy, which can mislead comparisons if not normalized by macro-F1.
- Human performance is reported as a lower bound because it is measured with single annotators, whereas ensembles of annotators perform significantly better.

## Evidence (verbatim from paper)

> Under reasoning-required setting, the task becomes much harder, but it's still possible for humans to solve: a single annotator can get 78.0% accuracy and 72.2% macro-F1.

## Citation

```bibtex
@misc{jin2019pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin et al. (2019)},
  year={2019},
  note={arXiv:1909.06146}
}
```

- arXiv: 1909.06146