Document Parsing Eval

---
name: document-parsing-eval
description: Evaluates end-to-end document parsing models on their ability to extract structured content (text, formulas, tables, reading order) from both standardized printed documents and real-world captured images. It measures structural fidelity, multilingual robustness, and decoding stability under visual degradation. Use when the user wants to benchmark on OmniDocBench, XFUND, Wild-OmniDocBench, or asks about evaluating this task. Reports Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.23885
  bibtex_key: li2026dochumming
  confidence: high
---

# document-parsing-eval

> Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training — Li et al. (2026) (arXiv:2603.23885, 2026)

## What this evaluates

Evaluates end-to-end document parsing models on their ability to extract structured content (text, formulas, tables, reading order) from both standardized printed documents and real-world captured images. It measures structural fidelity, multilingual robustness, and decoding stability under visual degradation.

## Datasets

- **OmniDocBench** — total ?; splits: test (-1)
- **XFUND** — total ?; splits: test (-1)
- **Wild-OmniDocBench** — total ?; splits: test (-1)

## Metrics

- `Overall` **(primary)** — range: percent
  - Composite score aggregating performance across text, formula, table, and reading-order sub-metrics. Higher is better.
- `TextEdit` — range: other
  - Edit distance for extracted text content. Lower is better.
- `FormulaCDM` — range: percent
  - Character/Token accuracy for mathematical formulas. Higher is better.
- `TableTEDS` — range: percent
  - Tree Edit Distance based Similarity for table structure. Higher is better.
- `Reading OrderEdit` — range: other
  - Edit distance for reading order of document elements. Lower is better.
- `Repeat` — range: [0, 1]
  - Fraction of outputs containing an identical structured pattern repeated more than 10 times and reaching maximum generation length. Lower is better.

## Input / output format

**Input**: Document images (scanned or digital) with varying layouts, illumination, and geometric distortions.

**Output**: Structured document representation (e.g., LaTeX or tokenized structure) generated autoregressively.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    text_edit = edit_distance(predictions.text, gold.text)
    formula_acc = token_accuracy(predictions.formulas, gold.formulas)
    table_teds = tree_edit_similarity(predictions.tables, gold.tables)
    order_edit = edit_distance(predictions.reading_order, gold.reading_order)
    overall = aggregate(text_edit, formula_acc, table_teds, order_edit)
    repeat_rate = sum(1 for p in predictions if is_repetitive(p)) / len(predictions)
    return {'TextEdit': text_edit, 'FormulaCDM': formula_acc, 'TableTEDS': table_teds, 'Reading OrderEdit': order_edit, 'Overall': overall, 'Repeat': repeat_rate}
```

## Common pitfalls

- TextEdit and Reading OrderEdit are error metrics where lower values indicate better performance, opposite to accuracy-based metrics.
- The 'Overall' score is a composite aggregate rather than a single standardized metric; direct comparison requires checking the aggregation method.
- Repetition rate is strictly defined by pattern repetition >10 times AND reaching max generation length, not just any repeated text.

## Evidence (verbatim from paper)

> Comparing #1 with #4, our method delivers clear gains on both settings: +3.79 Overall on OmniDocBench and +8.21 on Wild-OmniDocBench, while the repetition rate drops from 4.7→2.1 and 8.6→4.3, respectively.

## Citation

```bibtex
@misc{li2026dochumming,
  title={Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.23885}
}
```

- arXiv: 2603.23885