Litqa2 Eval

---
name: litqa2-eval
description: Evaluates an agentic system's ability to retrieve relevant scientific literature, re-rank passages, and answer multiple-choice questions based on the retrieved text. It probes retrieval coverage, passage localization, and question-answering accuracy under information loss constraints. Use when the user wants to benchmark on LitQA2, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.08752
  bibtex_key: [REDACTED]
  confidence: high
---

# litqa2-eval

> Patience is all you need! An agentic system for performing scientific literature review — Brett et al. (2025) (arXiv:2504.08752, 2025)

## What this evaluates

Evaluates an agentic system's ability to retrieve relevant scientific literature, re-rank passages, and answer multiple-choice questions based on the retrieved text. It probes retrieval coverage, passage localization, and question-answering accuracy under information loss constraints.

## Datasets

- **LitQA2** — total 199; splits: test (199)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Accuracy = Total correct answers / Total questions.
- `Coverage` — range: [0, 1]
  - Coverage = Total attempted answers / Total questions.
- `Precision` — range: [0, 1]
  - Precision = Total correct answers / Total attempted answers.

## Input / output format

**Input**: A multiple-choice question with answer options, and a set of retrieved scientific articles or text chunks.

**Output**: A single selected answer option from the provided choices.

## Scoring recipe

```python
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
attempted = len(predictions)
total = len(golds)
accuracy = correct / total
coverage = attempted / total
precision = correct / attempted
```

## Common pitfalls

- Evaluation is restricted to the PubMed Central Open Access subset (103/199 questions) due to full-text availability constraints.
- The benchmark assumes a single source paper contains the answer, but the authors find this assumption often fails in their corpus.
- Metrics are reported with 95% confidence intervals over 3 repeats, not just point estimates.

## Evidence (verbatim from paper)

> For evaluating the retrieval on full text articles in the PubMed Central Open Access dataset we can only consider a subset of the LitQA2 benchmark*[[12]]* (for which only 103 out of 199 are found in the PubMed Central OA subset). ... In the assessment of the benchmark we follow the same conventions as in the original paper*[[12]]*: Accuracy = Total correct answers / Total questions, Coverage = Total attempted answers / Total questions, Precision = Total correct answers / Total attempted answers

## Citation

```bibtex
@misc{brett2025patience,
  title={Patience is all you need! An agentic system for performing scientific literature review},
  author={Brett et al. (2025)},
  year={2025},
  note={arXiv:2504.08752}
}
```

- arXiv: 2504.08752