Olmocr Eval

---
name: olmocr-eval
description: Evaluates the ability of vision-language models and OCR tools to accurately linearize and extract structured content from complex, real-world PDFs. It probes reading order preservation, content comprehensiveness, and faithful representation of tables and equations. Use when the user wants to benchmark on olmOCR-mix-0225, or asks about evaluating this task. Reports alignment.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.18443
  bibtex_key: [REDACTED]
  confidence: high
---

# olmocr-eval

> olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models — Poznanski et al. (2025) (arXiv:2502.18443, 2025)

## What this evaluates

Evaluates the ability of vision-language models and OCR tools to accurately linearize and extract structured content from complex, real-world PDFs. It probes reading order preservation, content comprehensiveness, and faithful representation of tables and equations.

## Datasets

- **olmOCR-mix-0225** — total ?; splits: eval (2017); repo https://github.com/allenai/olmocr

## Metrics

- `alignment` **(primary)** — range: [0, 1]
  - Splits document into words, aligns them using Hirschberg’s algorithm, and calculates the proportion of matching words between the model output and the gold reference.
- `ELO rating` — range: ELO scale (base 1500)
  - Pairwise comparison score starting from a base of 1500. Computed as the average over 100 simulations to avoid ordering effects, with 95% confidence intervals via bootstrapping (5000 resamples).
- `win rate` — range: percent
  - Percentage of pairwise head-to-head comparisons where a model's output is preferred over another's based on human judgment.

## Input / output format

**Input**: PDF page images (or rendered pages) paired with plain text outputs from competing linearization tools for side-by-side comparison; or raw PDF pages for automated alignment scoring.

**Output**: Linearized text representation of the PDF page (e.g., Markdown, LaTeX, or plain text) preserving reading order and structured elements.

## Scoring recipe

```python
def calc_alignment(gold, pred):
    g_words = tokenize(gold)
    p_words = tokenize(pred)
    aligned = hirschberg_align(g_words, p_words)
    return sum(1 for g, p in aligned if g == p) / len(g_words)

def calc_elo(judgments, base=1500, sims=100):
    ratings = {m: base for m in models}
    for _ in range(sims):
        for pair in judgments:
            ratings[pair.winner] += 32 * (1 - expected_score(ratings[pair.winner], ratings[pair.loser]))
    return np.mean([run_elo_sim(judgments) for _ in range(sims)])

def calc_win_rate(wins, losses):
    return wins / (wins + losses) * 100
```

## Common pitfalls

- Alignment metric only measures word-level matching via Hirschberg's algorithm and ignores formatting differences (e.g., Markdown vs LaTeX).
- Human evaluation explicitly prioritizes 'usefulness for training LMs' over 'most faithful OCR', which may favor cleaner but slightly lossy outputs.
- ELO ratings are averaged over 100 simulations to mitigate ordering effects, so raw win rates alone do not reflect final rankings.

## Evidence (verbatim from paper)

> We calculate ELO ratings starting from a base of 1500 and report the average of 100 simulations to avoid ordering effects in ELO calculations; for 95% confidence intervals, we use bootstrapping with 5000 resamples.

## Citation

```bibtex
@misc{poznanski2025olmocr,
  title={olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models},
  author={Poznanski et al. (2025)},
  year={2025},
  note={arXiv:2502.18443}
}
```

- arXiv: 2502.18443