Paperqa Eval

---
name: paperqa-eval
description: Evaluates the accuracy and interaction efficiency of LLM agents performing multi-turn tool-use for scientific paper question-answering. It probes the model's ability to plan, execute tool calls, and extract answers from complex academic documents without excessive interaction or getting stuck in loops. Use when the user wants to benchmark on AirQA-Real, SciDQA, or asks about evaluating this task. Reports Avg., I-Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.12988
  bibtex_key: [REDACTED]
  confidence: high
---

# paperqa-eval

> PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient — Wang et al. (2026) (arXiv:2601.12988, 2026)

## What this evaluates

Evaluates the accuracy and interaction efficiency of LLM agents performing multi-turn tool-use for scientific paper question-answering. It probes the model's ability to plan, execute tool calls, and extract answers from complex academic documents without excessive interaction or getting stuck in loops.

## Datasets

- **AirQA-Real** — total ?; splits: test (-1)
- **SciDQA** — total ?; splits: test (-1)

## Metrics

- `Avg.` **(primary)** — range: percent
  - Average accuracy across sub-task modalities (text, table, image, form, meta). Calculated as the mean of correct answers per query.
- `I-Avg` **(primary)** — range: percent
  - I-Avg = Avg * sqrt(1 - I_bar / I_max), where I_bar is the average number of interaction turns and I_max is the maximum allowed interaction turns. Jointly measures performance and efficiency.
- `Valid Answers` — range: percent
  - Proportion of trials that successfully conclude within the maximum interaction limit without entering repetitive, non-productive loops.
- `Interaction Turns` — range: other
  - Average number of tool calls or interaction steps taken per query before submitting a final answer or hitting the limit.

## Input / output format

**Input**: Scientific paper QA questions requiring multi-turn tool use. Agents receive access to academic documents, tables, images, and forms via tool-use APIs to search and retrieve information.

**Output**: A sequence of tool-use actions (interactions) followed by a final answer. The protocol tracks the exact number of interaction turns until the answer is submitted or the limit is reached.

## Scoring recipe

```python
def compute_metrics(predictions, golds, interaction_turns, max_turns=100):
    accuracies = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    avg_acc = sum(accuracies) / len(accuracies)
    avg_turns = sum(interaction_turns) / len(interaction_turns)
    i_avg = avg_acc * (1 - (avg_turns / max_turns)) ** 0.5
    valid = sum(1 for t in interaction_turns if t <= max_turns) / len(interaction_turns)
    return {'Avg': avg_acc, 'I-Avg': i_avg, 'Valid Answers': valid, 'Interaction Turns': avg_turns}
```

## Common pitfalls

- I-Avg jointly optimizes accuracy and efficiency; optimizing solely for Avg may lead to inefficient, high-turn agents that score poorly on I-Avg.
- Avg aggregates performance across diverse modalities (text, table, image, form, meta), potentially hiding severe weaknesses in specific data types.
- The knowing-doing gap probe uses a separate UCB bandit task, not the main benchmarks, so results there are not directly comparable to Paper-QA scores.

## Evidence (verbatim from paper)

> Beyond the default metrics provided with each benchmark, we inspired from (Liu et al., [2025]), introducing an additional evaluation metric, I-Avg, defined as: I-Avg = Avg * sqrt(1 - I_bar / I_max), where I_bar is the average number of interaction turns, and I_max denotes the maximum number of interaction turns. The I-Avg metric is designed to provide a balanced assessment of agent behavior by jointly considering both its performance and efficiency.

## Citation

```bibtex
@misc{wang2026paperguide,
  title={PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficient},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.12988}
}
```

- arXiv: 2601.12988