Ocrbench Eval

---
name: ocrbench-eval
description: Evaluates Large Multimodal Models on five text-related visual tasks: text recognition, scene text-centric VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition. It probes the models' ability to perform precise visual pattern matching versus relying on semantic context, especially under challenging conditions like handwriting, multilingual text, blur, and complex layouts. Use when the user wants to benchmark on OCRBench, or asks about evaluating this task. Reports gt_present_in_output.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.07895
  bibtex_key: liu2023ocrbench
  confidence: high
---

# ocrbench-eval

> OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models — Liu et al. (2023) (arXiv:2305.07895, 2023)

## What this evaluates

Evaluates Large Multimodal Models on five text-related visual tasks: text recognition, scene text-centric VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition. It probes the models' ability to perform precise visual pattern matching versus relying on semantic context, especially under challenging conditions like handwriting, multilingual text, blur, and complex layouts.

## Datasets

- **OCRBench** — total 1000; splits: test (1000); repo https://github.com/Yuliang-Liu/MultimodalOCR

## Metrics

- `gt_present_in_output` **(primary)** — range: [0, 1]
  - Accuracy calculated as the fraction of instances where the ground truth string is a substring of the model's generated output. Questions with ground truth answers containing fewer than 4 symbols are filtered out prior to evaluation.

## Input / output format

**Input**: Image and a task-specific text prompt (e.g., 'what is written in the image?', 'What is the value for ‘{key}’?', or 'Please write out the expression of the formula in the image using LaTeX format.').

**Output**: Free-form text response.

## Scoring recipe

```python
def compute_metric(predictions, golds):
    valid = [(p, g) for p, g in zip(predictions, golds) if len(g) >= 4]
    if not valid:
        return 0.0
    correct = sum(1 for p, g in valid if g in p)
    return correct / len(valid)
```

## Common pitfalls

- Filtering out ground truth answers with fewer than 4 symbols can drastically reduce the effective test set size for short-answer tasks.
- The substring matching criterion may yield false positives if the model's verbose output accidentally contains the ground truth string as part of a longer phrase.
- Models often rely on semantic guessing rather than visual OCR, which the benchmark explicitly probes but the metric does not differentiate.

## Evidence (verbatim from paper)

> We have defined a unified and simple evaluation criterion for all datasets, which is to determine whether the ground truth (GT) is present in the output of the LMM. To reduce false positives, we filter out questions that have answers containing fewer than 4 symbols from all datasets.

## Citation

```bibtex
@misc{liu2023ocrbench,
  title={OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2305.07895}
}
```

- arXiv: 2305.07895