Olmocr Bench Eval

---
name: olmocr-bench-eval
description: Evaluates end-to-end OCR models on full-page transcription accuracy across diverse document types (academic papers, old scans, math, tables, multi-column layouts) and tests their ability to localize embedded images via bounding box prediction. Use when the user wants to benchmark on OlmOCR-Bench, LightOnOCR-bbox-bench, or asks about evaluating this task. Reports Overall Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14251
  bibtex_key: [REDACTED]
  confidence: high
---

# olmocr-bench-eval

> LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR — Taghadouini et al. (2026) (arXiv:2601.14251, 2026)

## What this evaluates

Evaluates end-to-end OCR models on full-page transcription accuracy across diverse document types (academic papers, old scans, math, tables, multi-column layouts) and tests their ability to localize embedded images via bounding box prediction.

## Datasets

- **OlmOCR-Bench** — total 1403; splits: test (1403)
- **LightOnOCR-bbox-bench** — total 855; splits: test (855)

## Metrics

- `Overall Score` **(primary)** — range: percent
  - Average character/token accuracy across document categories (ArXiv, Old Scans, Math, Tables, Multi-column, Long, Tiny Text). Headers/footers are explicitly excluded from scoring to align with full-page transcription objectives.
- `F1@0.5` — range: [0, 1]
  - F1 score for bounding box detection computed at an Intersection over Union (IoU) threshold of 0.5.
- `IoU` — range: [0, 1]
  - Mean Intersection over Union between predicted and ground truth bounding boxes for visual elements.
- `Count Accuracy` — range: percent
  - Percentage of pages where the model correctly predicts the exact number of visual elements (figures/images).
- `Throughput (pages/sec)` — range: other
  - Total number of pages processed divided by the wall-clock inference time to complete the benchmark.

## Input / output format

**Input**: High-resolution document images (single pages)

**Output**: Transcribed text (and bounding box coordinates for localization variants)

## Scoring recipe

```python
def compute_ocr_score(predictions, gold):
    cat_scores = []
    for cat in ['ArXiv', 'Old Scans', 'Math', 'Tables', 'Multi-column', 'Long', 'Tiny Text']:
        cat_scores.append(accuracy_score(predictions[cat], gold[cat]))
    return mean(cat_scores)  # Excludes headers/footers

def compute_bbox_metrics(pred_boxes, gt_boxes):
    f1 = f1_score_at_iou(pred_boxes, gt_boxes, iou_thresh=0.5)
    iou = mean_iou(pred_boxes, gt_boxes)
    count_acc = accuracy_score(len(pred_boxes), len(gt_boxes))
    return f1, iou, count_acc
```

## Common pitfalls

- The original OlmOCR-Bench definition rewards omitting headers/footers, which conflicts with full-page transcription objectives; the authors explicitly exclude this category from scoring.
- Evaluation is strictly single-pass without test-time heuristics like rotation sweeps or retries, so results reflect raw model capability rather than post-processed performance.

## Evidence (verbatim from paper)

> We evaluate LightOnOCR-2-1B on OlmOCR-Bench as our primary OCR benchmark. We evaluate all models without test-time heuristics (e.g., retries or rotation correction) to reflect raw model behavior. Table 1 shows that LightOnOCR-2-1B achieves the highest overall score ($83.2\pm 0.9$) among evaluated systems, outperforming substantially larger end-to-end models while using only 1B parameters and being end-to-end trainable.

## Citation

```bibtex
@misc{taghadouini2026lightonocr,
  title={LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},
  author={Taghadouini et al. (2026)},
  year={2026},
  note={arXiv:2601.14251}
}
```

- arXiv: 2601.14251