Vidore V3 Eval

---
name: vidore-v3-eval
description: This benchmark evaluates end-to-end Retrieval Augmented Generation (RAG) systems on visually rich, real-world documents across multiple professional domains. It probes a model's ability to retrieve relevant pages, generate accurate answers to complex open-ended and multi-hop queries, and precisely ground those answers with bounding boxes in multimodal content. Use when the user wants to benchmark on ViDoRe V3, or asks about evaluating this task. Reports F1 score (Dice coefficient).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08620
  bibtex_key: loison2026vidorev3
  confidence: high
---

# vidore-v3-eval

> ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios — Loison et al. (2026) (arXiv:2601.08620, 2026)

## What this evaluates

This benchmark evaluates end-to-end Retrieval Augmented Generation (RAG) systems on visually rich, real-world documents across multiple professional domains. It probes a model's ability to retrieve relevant pages, generate accurate answers to complex open-ended and multi-hop queries, and precisely ground those answers with bounding boxes in multimodal content.

## Datasets

- **ViDoRe V3** — total ?; splits: public (-1), private (-1)

## Metrics

- `F1 score (Dice coefficient)` **(primary)** — range: [0, 1]
  - Pixel-level overlap metric computed by merging all bounding boxes for a query into a single zone per annotator or model, then comparing zones across annotators. Averaged over pairwise comparisons when multiple annotators label the same sample.
- `Intersection over Union (IoU)` — range: [0, 1]
  - Ratio of the intersection area to the union area of two bounding box zones, used alongside F1 to measure grounding precision and recall trade-offs.
- `Gwet’s AC2` — range: [-1, 1]
  - Statistical measure of inter-annotator agreement that remains stable under prevalence skew, used to validate the quality of query-page relevance labeling.

## Input / output format

**Input**: Text query (question, keyword, or instruction) paired with visually rich document pages containing text, tables, charts, infographics, and images.

**Output**: Retrieved relevant pages, a generated answer based on those pages, and bounding box coordinates grounding the answer within the document images.

## Scoring recipe

```python
def compute_grounding_metrics(pred_boxes, gold_boxes):
    # Merge boxes into zones if multiple per query
    pred_zone = merge_boxes(pred_boxes)
    gold_zone = merge_boxes(gold_boxes)
    iou = intersection_over_union(pred_zone, gold_zone)
    # F1/Dice is derived from IoU or computed via pixel overlap
    f1 = 2 * iou / (iou + 1) if iou > 0 else 0
    return {'IoU': iou, 'F1': f1}
```

## Common pitfalls

- Inter-annotator agreement is moderate (IoU 0.50, F1 0.60), so model performance should be evaluated against human upper bounds rather than assuming perfect gold standards.
- The benchmark uses a split-release strategy with 8 public and 2 private corpora to prevent data contamination; evaluating on private sets requires blind submission protocols.
- Queries often require interpreting non-textual elements (charts, infographics, tables), which current vision-language models frequently fail to ground or reason over accurately.

## Evidence (verbatim from paper)

> We compute inter-annotator agreement on the subset of query-page pairs labeled by two or three annotators. For each annotator, we merge all their bounding boxes into a single zone. We then compare zones across annotators by measuring pixel-level overlap, reporting Intersection over Union (IoU) and F1 score (Dice coefficient). When three annotators label the same sample, we average over all pairwise comparisons.

## Citation

```bibtex
@misc{loison2026vidorev3,
  title={ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios},
  author={Loison et al. (2026)},
  year={2026},
  note={arXiv:2601.08620}
}
```

- arXiv: 2601.08620