Varex Eval

---
name: varex-eval
description: Evaluates multi-modal structured data extraction from documents, testing a model's ability to parse visual or textual layouts, adhere to a provided JSON schema, and generate compliant structured outputs. It specifically probes schema compliance, layout understanding, and cross-modal robustness across plain text, spatial text, and image inputs. Use when the user wants to benchmark on VAREX, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15118
  bibtex_key: barzelay2026varex
  confidence: high
---

# varex-eval

> VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents — Barzelay et al. (2026) (arXiv:2603.15118, 2026)

## What this evaluates

Evaluates multi-modal structured data extraction from documents, testing a model's ability to parse visual or textual layouts, adhere to a provided JSON schema, and generate compliant structured outputs. It specifically probes schema compliance, layout understanding, and cross-modal robustness across plain text, spatial text, and image inputs.

## Datasets

- **VAREX** — total 1777; splits: test (1777); repo https://github.com/udibarzi/varex-bench

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - A field scores 1 if the normalized prediction exactly matches the normalized ground truth, and 0 otherwise. For array fields, order-invariant matching is applied via the Hungarian algorithm to assign predicted elements to ground-truth elements by maximum overlap.
- `ANLS (Average Normalized Levenshtein Similarity)` — range: [0, 1]
  - Assigns partial credit for near-matches by computing the normalized Levenshtein similarity between the normalized prediction and ground truth, distinguishing complete misses from minor formatting differences.

## Input / output format

**Input**: A minimal zero-shot prompt instructing the model to extract structured data matching a provided schema and return valid JSON. Inputs are provided in one of four modalities: Plain Text (P), Spatial Text (S), Image (V at 200 or 50 DPI), or Spatial Text + Image (S+V).

**Output**: Valid JSON object matching the provided schema, with null values for missing fields. Models are instructed to use response_format: {"type": "json_object"}.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    em_scores = []
    anls_scores = []
    for pred, gt in zip(predictions, ground_truth):
        pred_norm = normalize_text(pred)
        gt_norm = normalize_text(gt)
        if pred_norm == gt_norm:
            em_scores.append(1.0)
            anls_scores.append(1.0)
        else:
            em_scores.append(0.0)
            anls_scores.append(normalized_levenshtein_similarity(pred_norm, gt_norm))
    return {
        'EM': sum(em_scores) / len(em_scores),
        'ANLS': sum(anls_scores) / len(anls_scores)
    }
```

## Common pitfalls

- Schema echo: Models below 4B parameters often fail on output compliance/formatting rather than actual extraction, causing 45–65 percentage point score drops.
- Array ordering: Models may traverse table rows in different orders; failing to use order-invariant matching (Hungarian algorithm) unfairly penalizes correct extractions.
- Modality bias: Performance varies significantly across P, S, V, and S+V inputs; comparing scores across modalities without accounting for layout preservation or DPI differences can misrepresent model capability.

## Evidence (verbatim from paper)

> We report exact match (EM) as the primary metric: a field scores 1 if the normalized prediction exactly matches the normalized ground truth, 0 otherwise. We additionally report ANLS (Average Normalized Levenshtein Similarity), which assigns partial credit for near-matches, to distinguish complete misses from minor formatting differences. For array fields, we apply order-invariant matching via the Hungarian algorithm: predicted array elements are optimally assigned to ground-truth elements by maximum field overlap rather than positional index, ensuring models are not penalized for reading table rows in a different traversal order.

## Citation

```bibtex
@misc{barzelay2026varex,
  title={VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents},
  author={Barzelay et al. (2026)},
  year={2026},
  note={arXiv:2603.15118}
}
```

- arXiv: 2603.15118