Cogdoc Eval

---
name: cogdoc-eval
description: Evaluates vision-language models on multi-page document understanding, specifically testing long-context compositional reasoning, fine-grained information extraction from forms, complex layout and chart comprehension, and cross-page navigation for answer localization. Use when the user wants to benchmark on MMLongbench-Doc, DUDE, SlideVQA, MP-DocVQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.12658
  bibtex_key: xu2025cogdoc
  confidence: high
---

# cogdoc-eval

> CogDoc: Towards Unified thinking in Documents — Qixin Xu et al. (2025) (arXiv:2512.12658, 2025)

## What this evaluates

Evaluates vision-language models on multi-page document understanding, specifically testing long-context compositional reasoning, fine-grained information extraction from forms, complex layout and chart comprehension, and cross-page navigation for answer localization.

## Datasets

- **MMLongbench-Doc** — total ?; splits: test (-1)
- **DUDE** — total ?; splits: test (-1)
- **SlideVQA** — total ?; splits: test (-1)
- **MP-DocVQA** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Exact match ratio: 1 if the predicted answer or retrieved page set exactly matches the ground truth, 0 otherwise.
- `ANLS` — range: [0, 1]
  - Average Normalized Levenshtein Similarity between predicted and ground-truth text spans, normalized by the length of the longer string.
- `F1` — range: [0, 1]
  - Token-level F1 score computed between predicted and ground-truth answers.

## Input / output format

**Input**: Multi-page document images paired with natural language queries requiring reasoning, information extraction, or page navigation.

**Output**: Sequential two-stage output: first, a set of retrieved page identifiers; second, a final textual answer or extracted value.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    anls = sum(normalized_levenshtein(p, g) for p, g in zip(predictions, golds)) / len(golds)
    f1 = compute_token_f1(predictions, golds)
    return {'Accuracy': acc, 'ANLS': anls, 'F1': f1}
```

## Common pitfalls

- Assuming the SFT+RL training strategy universally outperforms Direct RL; the paper shows SFT+RL degrades performance on long-context benchmarks due to internal policy conflict.
- Treating MP-DocVQA results as in-distribution performance; the model was explicitly not trained on its trainset, so scores measure zero-shot generalization to unseen document types.
- Confusing Stage 1 retrieval metrics (page ID accuracy/recall) with Stage 2 answer metrics (ANLS/Accuracy/F1) when comparing ablation variants.

## Evidence (verbatim from paper)

> Acc. (Accuracy) indicates if the retrieved page identifiers set is an exact match to the ground truth, while Rec. (Recall) indicates if the retrieved set fully includes all relevant page numbers.

## Citation

```bibtex
@misc{xu2025cogdoc,
  title={CogDoc: Towards Unified thinking in Documents},
  author={Qixin Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.12658}
}
```

- arXiv: 2512.12658