Papermind Eval

---
name: papermind-eval
description: Evaluates multimodal LLMs' ability to perform integrated agentic reasoning and critical assessment over scientific papers. It probes capabilities in multimodal grounding, experimental interpretation, cross-source evidence synthesis via tool use, and critical evaluation of research claims. Use when the user wants to benchmark on PaperMind, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21304
  bibtex_key: zhao2026papermind
  confidence: high
---

# papermind-eval

> PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs — Yanjun Zhao et al. (arXiv:2604.21304, 2026)

## What this evaluates

Evaluates multimodal LLMs' ability to perform integrated agentic reasoning and critical assessment over scientific papers. It probes capabilities in multimodal grounding, experimental interpretation, cross-source evidence synthesis via tool use, and critical evaluation of research claims.

## Datasets

- **PaperMind** — total ?; splits: test (-1); repo https://github.com/Yanjun-Zhao/PaperMind

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Token-level or entity-level F1 score computed as the harmonic mean of precision and recall between the model's generated answer and the ground truth.
- `LLM-as-a-Judge` — range: other
  - A 5-point scale rating generated by an LLM judge evaluating the quality and correctness of the model's response against the gold answer and context.
- `Average interaction steps` — range: other
  - Mean number of reasoning steps or turns the model takes to answer a question.
- `Average tool usage` — range: other
  - Mean number of external tool invocations per question during agentic reasoning tasks.

## Input / output format

**Input**: Multimodal scientific papers (text and figures) paired with task-specific questions. Inputs may optionally include the paper's introduction as background context, and for agentic tasks, the model receives a query requiring external evidence retrieval.

**Output**: Free-form text answers to questions, and for agentic tasks, structured tool usage traces (invocations and results) generated via the smolagents/ReAct framework.

## Scoring recipe

```python
def compute_f1(pred, gold):
    pred_tokens = set(pred.split())
    gold_tokens = set(gold.split())
    tp = len(pred_tokens & gold_tokens)
    fp = len(pred_tokens - gold_tokens)
    fn = len(gold_tokens - pred_tokens)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_llm_judge(pred, gold, ctx):
    prompt = f'Judge answer: {pred} vs gold: {gold} in context: {ctx}'
    return llm_judge_model(prompt) # returns 1-5
```

## Common pitfalls

- Performance is highly sensitive to the maximum tool call budget; exceeding 6-8 steps often degrades performance due to context dilution.
- F1 scores and LLM-as-a-Judge ratings can diverge, meaning high lexical overlap does not guarantee high qualitative judgment scores.
- Models tend to over-rely on general web search rather than domain-specific tools (e.g., arXiv retriever), which may skew cross-domain evaluation results.

## Evidence (verbatim from paper)

> We evaluate model performance using both F1 score and an LLM-as-a-Judge metric on a 5-point scale.

## Citation

```bibtex
@misc{zhao2026papermind,
  title={PAPERMIND: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs},
  author={Yanjun Zhao et al.},
  year={2026},
  note={arXiv:2604.21304}
}
```

- arXiv: 2604.21304