Qianfan Ocr Eval

---
name: qianfan-ocr-eval
description: This evaluation probes a unified vision-language model's ability to perform end-to-end document intelligence, including specialized OCR, general text recognition, document understanding, and key information extraction across diverse document types and multilingual scenarios. Use when the user wants to benchmark on Omni-Doc-Bench v1.5, OLMOCRBench, OCRBench, DocVQA, ChartQA, Nanonets KIE, or asks about evaluating this task. Reports normalized accuracy (0-100).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13398
  bibtex_key: dong2026qianfanocr
  confidence: high
---

# qianfan-ocr-eval

> Qianfan-OCR: A Unified End-to-End Model for Document Intelligence — Daxiang Dong et al. (arXiv:2603.13398, 2026)

## What this evaluates

This evaluation probes a unified vision-language model's ability to perform end-to-end document intelligence, including specialized OCR, general text recognition, document understanding, and key information extraction across diverse document types and multilingual scenarios.

## Datasets

- **Omni-Doc-Bench v1.5** — total ?; splits: (unstated)
- **OLMOCRBench** — total ?; splits: (unstated)
- **OCRBench** — total ?; splits: (unstated)
- **DocVQA** — total ?; splits: (unstated)
- **ChartQA** — total ?; splits: (unstated)
- **Nanonets KIE** — total ?; splits: (unstated)

## Metrics

- `F1 score` — range: percent
  - Harmonic mean of precision and recall for extracted key-value fields, calculated as 2 * (precision * recall) / (precision + recall).
- `normalized accuracy (0-100)` **(primary)** — range: percent
  - Raw accuracy or F1 scores are linearly scaled to a 0–100 range for consistent cross-benchmark comparison.

## Input / output format

**Input**: Document images, optionally accompanied by text prompts, questions, or layout-as-thought triggers (⟨think⟩ tokens).

**Output**: Markdown-formatted text, structured layout representations (bounding boxes, element types, reading order), or extracted key-value fields depending on the benchmark.

## Scoring recipe

```python
def score_kie(predictions, gold):
    pred_fields = extract_fields(predictions)
    gold_fields = extract_fields(gold)
    tp = len(set(pred_fields) & set(gold_fields))
    precision = tp / len(pred_fields) if pred_fields else 0
    recall = tp / len(gold_fields) if gold_fields else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100  # Normalize to 0-100 scale
```

## Common pitfalls

- Comparing end-to-end VLMs directly against specialized pipeline OCR systems without accounting for architectural differences and parameter counts.
- Assuming KIE scores are raw F1 without applying the paper's specified 0–100 normalization scale.

## Evidence (verbatim from paper)

> Nanonets KIE (real-world invoice and receipt parsing, measured by F1 score). All scores are normalized to a 0–100 scale.

## Citation

```bibtex
@misc{dong2026qianfanocr,
  title={Qianfan-OCR: A Unified End-to-End Model for Document Intelligence},
  author={Daxiang Dong et al.},
  year={2026},
  note={arXiv:2603.13398}
}
```

- arXiv: 2603.13398