Unidoc Bench Eval

---
name: unidoc-bench-eval
description: Evaluates retrieval and end-to-end generation performance of multimodal RAG systems on real-world PDF documents. It probes the ability of text-only, image-only, and multimodal (text-image fusion/joint) retrieval paradigms to locate relevant evidence and generate faithful, complete answers to cross-modality questions. Use when the user wants to benchmark on UNIDOC-BENCH, or asks about evaluating this task. Reports Precision@10, Recall@10, Faithfulness, Completeness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.03663
  bibtex_key: [REDACTED]
  confidence: high
---

# unidoc-bench-eval

> UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG — Peng et al. (2025) (arXiv:2510.03663, 2025)

## What this evaluates

Evaluates retrieval and end-to-end generation performance of multimodal RAG systems on real-world PDF documents. It probes the ability of text-only, image-only, and multimodal (text-image fusion/joint) retrieval paradigms to locate relevant evidence and generate faithful, complete answers to cross-modality questions.

## Datasets

- **UNIDOC-BENCH** — total 1600; splits: test (1600); repo https://github.com/SalesforceAIResearch/UniDOC-Bench

## Metrics

- `Precision@10` **(primary)** — range: [0, 1]
  - Fraction of the top-10 retrieved candidates that match the ground-truth context in both page number and file. A true positive requires exact page and file match.
- `Recall@10` **(primary)** — range: [0, 1]
  - Fraction of ground-truth contexts successfully retrieved within the top-10 candidates. Partial page overlaps are counted as true positives.
- `Faithfulness` **(primary)** — range: [0, 1]
  - LLM-based metric measuring whether the facts extracted from the system's response are grounded in the ground-truth chunks. Calculated as the proportion of response facts verified against ground-truth evidence.
- `Completeness` **(primary)** — range: [0, 1]
  - LLM-based metric measuring whether the facts required to answer the question (extracted from the ground-truth answer) appear in the system's response. Calculated as the proportion of ground-truth facts present in the response.

## Input / output format

**Input**: Per instance: a question, a set of PDF documents (provided as text chunks and/or page-level JPEG images), and ground-truth answer/context metadata.

**Output**: Per instance: top-10 retrieved candidates (text chunks or images) for retrieval evaluation; a final generated text response for end-to-end evaluation.

## Scoring recipe

```python
def score_retrieval(preds, gold):
    tp = sum(1 for p in preds[:10] if p.page == gold.page and p.file == gold.file)
    return tp / 10, tp / len(gold)

def score_e2e(response, gold_answer, gold_chunks, llm_judge):
    resp_facts = llm_judge.extract_facts(response)
    gold_facts = llm_judge.extract_facts(gold_answer)
    faithfulness = len([f for f in resp_facts if f in gold_chunks]) / len(resp_facts)
    completeness = len(set(resp_facts) & set(gold_facts)) / len(gold_facts)
    return faithfulness, completeness
```

## Common pitfalls

- Partial page overlaps (e.g., retrieved pages 1–3 vs. ground-truth pages 3–5) are counted as true positives, which slightly inflates Recall@10 scores.
- Absolute metric values should not be overinterpreted; the benchmark is designed to highlight relative performance differences across retrieval paradigms rather than provide absolute capability baselines.
- Image-only retrieval often achieves higher recall at the retrieval stage but fails to translate to better end-to-end completeness due to LLM processing limitations on page-level images.

## Evidence (verbatim from paper)

> Specifically, we first ask the LLM to extract the facts required to answer each question and then verify whether these facts are grounded in the ground-truth chunks; this is measured as faithfulness. Next, we ask the LLM to extract the facts required to answer the question from the ground-truth answer and then check whether each fact appears in the system’s response; this is measured as completeness. Higher faithfulness and completeness scores are better.

## Citation

```bibtex
@misc{peng2025unidocbench,
  title={UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2510.03663}
}
```

- arXiv: 2510.03663