Omnidocbench Eval

---
name: omnidocbench-eval
description: Evaluates end-to-end document parsing capabilities, including text recognition, formula recognition, table structure extraction, and reading order prediction across diverse document types and languages. Use when the user wants to benchmark on OmniDocBench, or asks about evaluating this task. Reports OverallEdit.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.05218
  bibtex_key: zhang2025monkeyocr
  confidence: high
---

# omnidocbench-eval

> MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm — Zhang Li et al. (2025) (arXiv:2506.05218, 2025)

## What this evaluates

Evaluates end-to-end document parsing capabilities, including text recognition, formula recognition, table structure extraction, and reading order prediction across diverse document types and languages.

## Datasets

- **OmniDocBench** — total 981; splits: test (981)

## Metrics

- `OverallEdit` **(primary)** — range: [0, 1]
  - Character-level edit distance between the predicted and ground-truth parsed document text. Lower values indicate better alignment.
- `TextEdit` — range: [0, 1]
  - Character-level edit distance computed only on text blocks. Lower is better.
- `FormulaEdit` — range: [0, 1]
  - Character-level edit distance for formula regions. Lower is better.
- `FormulaCDM` — range: [0, 1]
  - Character-level Document Matching score measuring alignment accuracy for formulas. Higher is better.
- `TableTEDS` — range: [0, 1]
  - Tree Edit Distance based Similarity for table structure prediction. Higher is better.
- `TableEdit` — range: [0, 1]
  - Character-level edit distance for table content. Lower is better.
- `Read OrderEdit` — range: [0, 1]
  - Edit distance for the predicted reading order sequence compared to ground truth. Lower is better.

## Input / output format

**Input**: Single PDF pages (images) containing mixed content (text, formulas, tables, figures).

**Output**: Parsed document structure including recognized text, LaTeX/formula representations, table structures, and reading order sequence.

## Scoring recipe

```python
def compute_metrics(pred, gold):
    # Edit distances (lower is better)
    overall_edit = levenshtein_distance(pred.text, gold.text)
    text_edit = levenshtein_distance(pred.text_blocks, gold.text_blocks)
    formula_edit = levenshtein_distance(pred.formulas, gold.formulas)
    read_order_edit = levenshtein_distance(pred.reading_order, gold.reading_order)
    table_edit = levenshtein_distance(pred.tables_content, gold.tables_content)
    # Structure/Alignment scores (higher is better)
    formula_cdm = character_document_matching(pred.formulas, gold.formulas)
    table_teds = tree_edit_distance_similarity(pred.table_structure, gold.table_structure)
    return {
        'OverallEdit': overall_edit,
        'TextEdit': text_edit,
        'FormulaEdit': formula_edit,
        'FormulaCDM': formula_cdm,
        'TableTEDS': table_teds,
        'TableEdit': table_edit,
        'Read OrderEdit': read_order_edit
    }
```

## Common pitfalls

- Edit distance scores are highly sensitive to tokenization and formatting differences (e.g., spacing, line breaks), which can disproportionately penalize minor structural variations.
- Language-specific tokenization affects EN vs ZH scores; Chinese characters are typically evaluated at the character level, while English may use word-level tokenization, making cross-lingual comparison non-trivial.
- TableTEDS requires exact tree structure alignment; missing or merged cells in predictions drastically reduce scores even if visual layout appears correct.

## Evidence (verbatim from paper)

> OmniDocBench is a benchmark designed to evaluate real-world document parsing capabilities. It comprises 981 PDF pages spanning 9 document types, 4 layout styles, and 3 language categories. As shown in Table 2, MonkeyOCR achieves the best overall performance on both Chinese and English document parsing tasks. In particular, MonkeyOCR surpasses MinerU by over 6% in overall edit distance for Chinese documents, exceeds MinerU by an average of 15.0% in formula recognition across Chinese and English, and outperforms MinerU by 8.6% on average in table recognition for both languages.

## Citation

```bibtex
@misc{zhang2025monkeyocr,
  title={MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm},
  author={Zhang Li et al. (2025)},
  year={2025},
  note={arXiv:2506.05218}
}
```

- arXiv: 2506.05218