Document Haystack Eval

---
name: document-haystack-eval
description: This benchmark evaluates Vision Language Models' ability to retrieve specific textual or multimodal "needles" embedded within long documents ranging from 5 to 200 pages. It probes visual-text alignment, long-context retrieval capabilities, and performance degradation as document length and token consumption increase. Use when the user wants to benchmark on Document Haystack, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.15882
  bibtex_key: huybrechts2025documenthaystack
  confidence: high
---

# document-haystack-eval

> Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark — Huybrechts et al. (2025) (arXiv:2507.15882, 2025)

## What this evaluates

This benchmark evaluates Vision Language Models' ability to retrieve specific textual or multimodal "needles" embedded within long documents ranging from 5 to 200 pages. It probes visual-text alignment, long-context retrieval capabilities, and performance degradation as document length and token consumption increase.

## Datasets

- **Document Haystack** — total 400; splits: test (-1); repo https://github.com/amazon-science/document-haystack

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly retrieved needles out of the total evaluated. Calculated as (number of correct retrievals / total needles) * 100.

## Input / output format

**Input**: Long documents (5–200 pages) provided either as images or parsed text, accompanied by a natural language query asking to retrieve a specific embedded needle (e.g., “What is the secret fruit in the document?”).

**Output**: Extracted text or description corresponding to the queried needle. Exact formatting is not strictly enforced, but retrieval must match the ground truth needle.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- API token limits cause models like GPT-4o-mini to fail or truncate on documents exceeding 50 pages, artificially lowering reported accuracy.
- Preprocessing method (converting PDFs to images vs. extracting raw text) drastically changes performance; comparing models that use different preprocessing pipelines directly is misleading.
- Token compression strategies vary widely across models (e.g., Nova Lite uses ~6x more tokens per image than Gemini), which impacts context window utilization and retrieval accuracy independently of model capability.

## Evidence (verbatim from paper)

> The table shows the retrieval accuracy across different document lengths of three API providers: Nova Lite, Gemini Flash-2.0, and GPT-4o-mini.

## Citation

```bibtex
@misc{huybrechts2025documenthaystack,
  title={Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark},
  author={Huybrechts et al. (2025)},
  year={2025},
  note={arXiv:2507.15882}
}
```

- arXiv: 2507.15882