Vidoseek Eval

---
name: vidoseek-eval
description: Evaluates a multi-agent RAG framework's ability to retrieve relevant pages from visually rich documents and generate accurate answers through iterative reasoning. It probes hybrid visual-textual retrieval and dynamic token allocation for document comprehension. Use when the user wants to benchmark on ViDoSeek, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.18017
  bibtex_key: wang2025vidorag
  confidence: high
---

# vidoseek-eval

> ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents — Qiuchen Wang et al. (2025) (arXiv:2502.18017, 2025)

## What this evaluates

Evaluates a multi-agent RAG framework's ability to retrieve relevant pages from visually rich documents and generate accurate answers through iterative reasoning. It probes hybrid visual-textual retrieval and dynamic token allocation for document comprehension.

## Datasets

- **ViDoSeek** — total 6000; splits: test (-1); repo https://github.com/Alibaba-NLP/ViDoRAG

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Model-based evaluation using GPT-4o to score generated answers against references on a 1-5 scale. Accuracy is the proportion of answers scoring 4 or higher.
- `Recall@K` — range: percent
  - Standard recall at K=1, 3, 5 measuring the fraction of queries where the golden page appears in the top-K retrieved results. Also reports MRR@5.

## Input / output format

**Input**: Visual document images and natural language queries.

**Output**: Final generated answer (for generation task) or retrieved document/page IDs (for retrieval task).

## Scoring recipe

```python
# Generation Accuracy
scores = gpt4o_score(reference_answer, generated_answer)  # 1-5 scale
accuracy = sum(1 for s in scores if s >= 4) / len(scores)

# Retrieval Recall@K
recall_at_k = len(set(retrieved_top_k) & set(golden_pages)) / len(golden_pages)
```

## Common pitfalls

- GPT-4o scoring uses a threshold of 4/5 for correctness rather than exact string matching.
- Retrieval baselines are normalized to the average dynamic recall length of the proposed method to ensure fair comparison, rather than using a fixed top-K across all methods.

## Evidence (verbatim from paper)

> For our end-to-end evaluation, we employed a model-based assessment using GPT-4o, which involved assigning scores from 1 to 5 by comparing the reference answer with the final answer. Answers receiving scores of 4 or above were considered correct, and we subsequently calculate accuracy as the evaluation metric. For retrieval evaluation, we use recall as the metric.

## Citation

```bibtex
@misc{wang2025vidorag,
  title={ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents},
  author={Qiuchen Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.18017}
}
```

- arXiv: 2502.18017