Pdfqa Eval

---
name: pdfqa-eval
description: Evaluates end-to-end question answering over PDF documents, probing parsing, retrieval, and reasoning capabilities across diverse document types, modalities, and complexity dimensions. Use when the user wants to benchmark on pdfQA, or asks about evaluating this task. Reports G-Eval correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02285
  bibtex_key: schimanski2026pdfqa
  confidence: high
---

# pdfqa-eval

> pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs — Schimanski et al. (2026) (arXiv:2601.02285, 2026)

## What this evaluates

Evaluates end-to-end question answering over PDF documents, probing parsing, retrieval, and reasoning capabilities across diverse document types, modalities, and complexity dimensions.

## Datasets

- **pdfQA** — total 4000; splits: test (4000); repo https://github.com/tobischimanski/pdfQA

## Metrics

- `G-Eval correctness` **(primary)** — range: [1, 5]
  - LLM-as-judge evaluation using a G-Eval prompt that rates answer correctness on a 1–5 scale, where 5 indicates a completely correct answer.

## Input / output format

**Input**: Parsed text extracted from the source PDF using PyMUPDF, concatenated with the corresponding question.

**Output**: Free-form text answer generated by the model, which is then scored by an LLM judge.

## Scoring recipe

```python
def compute_metric(predictions, questions, contexts):
    scores = []
    for pred, q, ctx in zip(predictions, questions, contexts):
        prompt = f'Question: {q}\nContext: {ctx}\nAnswer: {pred}\nRate correctness 1-5:'
        score = llm_as_judge(prompt)  # Returns int 1-5
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- The evaluation relies entirely on an LLM-as-judge (G-Eval) rather than exact match or human verification, which can introduce scoring bias or inconsistency.
- PDF parsing is done naively by extracting all text via PyMUPDF without preserving layout or structure, potentially disadvantaging models that rely on visual or structural cues.
- Averaging scores across heterogeneous complexity dimensions (e.g., modality, file length, reasoning depth) can mask severe performance drops on specific subtasks like table interpretation or long-document retrieval.

## Evidence (verbatim from paper)

> Then, we use the G-Eval correctness prompt in Figure [8] to evaluate the answer. This prompt scales the answer between 1-5, where 5 indicates a completely correct answer (see App. [J]).

## Citation

```bibtex
@misc{schimanski2026pdfqa,
  title={pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs},
  author={Schimanski et al. (2026)},
  year={2026},
  note={arXiv:2601.02285}
}
```

- arXiv: 2601.02285