Mosaicdoc Eval

---
name: mosaicdoc-eval
description: Probes Visually Rich Document Understanding (VRDU) capabilities across three tasks: document VQA, page-level OCR, and reading order prediction. It specifically tests models' ability to comprehend dense, bilingual, non-Manhattan layouts, perform multi-span reasoning, and maintain global layout coherence without token reduction artifacts. Use when the user wants to benchmark on MosaicDoc, or asks about evaluating this task. Reports ANLSL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.09919
  bibtex_key: chen2025mosaicdoc
  confidence: high
---

# mosaicdoc-eval

> MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding — Chen et al. (2025) (arXiv:2511.09919, 2025)

## What this evaluates

Probes Visually Rich Document Understanding (VRDU) capabilities across three tasks: document VQA, page-level OCR, and reading order prediction. It specifically tests models' ability to comprehend dense, bilingual, non-Manhattan layouts, perform multi-span reasoning, and maintain global layout coherence without token reduction artifacts.

## Datasets

- **MosaicDoc** — total 72000; splits: test (-1); repo https://github.com/DOCLAB-SCUT/MosaicDoc

## Metrics

- `ANLSL` **(primary)** — range: [0, 1]
  - Average Normalized Levenshtein Similarity for List. Computes the normalized Levenshtein similarity between predicted and ground-truth answer lists, averaged across the dataset.
- `CRR` — range: percent
  - Character Recognition Rate. Measures character-level accuracy by comparing the model's full page transcription against the ground truth.
- `OCRR` — range: percent
  - Output-based Character Recognition Rate. Normalizes character accuracy by the length of the model's generated output to measure precision within the prediction.
- `Micro-F1` — range: [0, 1]
  - Micro-averaged F1 score computed on text line sequences to evaluate reading order prediction correctness by matching predicted blocks to the ground truth sequence.

## Input / output format

**Input**: Image of a visually rich document (magazine or newspaper) paired with a question (for VQA), or image alone (for OCR and reading order prediction).

**Output**: Text string (answer for VQA, full page transcription for OCR, ordered list of text line blocks for reading order prediction).

## Scoring recipe

```python
def compute_anlsl(pred_answers, gold_answers):
    sims = [levenshtein_similarity(p, g) for p, g in zip(pred_answers, gold_answers)]
    return sum(sims) / len(sims)

def compute_crr(pred_text, gold_text):
    return char_accuracy(pred_text, gold_text)

def compute_ocrr(pred_text, gold_text):
    return char_accuracy(pred_text, gold_text) / len(pred_text)

def compute_micro_f1(pred_order, gold_order):
    tp = sum(1 for p, g in zip(pred_order, gold_order) if p == g)
    fp = len(pred_order) - tp
    fn = len(gold_order) - tp
    return 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) > 0 else 0.0
```

## Common pitfalls

- Expert VLMs using token reduction/merging strategies lose critical semantic information on dense, information-rich layouts.
- Models frequently exhibit repetitive output generation until hitting token limits, which artificially deflates CRR/OCRR scores.
- Reading order prediction often yields high precision but low recall, as models correctly order local column fragments but fail to capture global multi-column or non-Manhattan sequencing.

## Evidence (verbatim from paper)

> We evaluate DocVQA performance using the Average Normalized Levenshtein Similarity for List (ANLSL) metric. To assess raw text recognition capability, we evaluate page-level Character Recognition Rate (OCR) using CRR and Output-based Character Recognition Rate (OCRR). We evaluate ROP using the Micro-F1 score on text line sequences.

## Citation

```bibtex
@misc{chen2025mosaicdoc,
  title={MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.09919}
}
```

- arXiv: 2511.09919