Docbank Layout Eval

---
name: docbank-layout-eval
description: Evaluates a model's ability to identify and classify semantic document structures (e.g., sections, figures, equations) from serialized 2D document pages. It probes multimodal layout understanding by measuring how well token-level predictions align with ground-truth semantic units, even when tokens are discontinuous. Use when the user wants to benchmark on DocBank, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.01038
  bibtex_key: li2020docbank
  confidence: high
---

# docbank-layout-eval

> DocBank: A Benchmark Dataset for Document Layout Analysis — Minghao Li et al. (2020) (arXiv:2006.01038, 2020)

## What this evaluates

Evaluates a model's ability to identify and classify semantic document structures (e.g., sections, figures, equations) from serialized 2D document pages. It probes multimodal layout understanding by measuring how well token-level predictions align with ground-truth semantic units, even when tokens are discontinuous.

## Datasets

- **DocBank** — total ?; splits: test (-1); repo https://github.com/doc-analysis/DocBank

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Precision = Area of Ground truth tokens in Detected tokens / Area of all Detected tokens. Recall = Area of Ground truth tokens in Detected tokens / Area of all Ground truth tokens. F1 Score = 2 * Precision * Recall / (Precision + Recall). Area is computed as the count of tokens in the respective sets.

## Input / output format

**Input**: Serialized sequence of tokens from a 2D document page, truncated to a maximum block size of 512. Includes layout/position features.

**Output**: Token-level classification labels for 12 semantic structure categories (Abstract, Author, Caption, Equation, Figure, Footer, List, Paragraph, Reference, Section, Table, Title).

## Scoring recipe

```python
def compute_f1(pred_tokens, gt_tokens):
    intersection = len(set(pred_tokens) & set(gt_tokens))
    precision = intersection / len(pred_tokens) if pred_tokens else 0
    recall = intersection / len(gt_tokens) if gt_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
# Compute per category, then macro-average across all 12 semantic structures
```

## Common pitfalls

- Standard BIO-tagging evaluation fails because tokens belonging to the same semantic unit can be discontinuous in the serialized sequence.
- Metrics are computed per semantic category, not as a single global accuracy; results must be macro-averaged across all 12 structure types.
- Vision models output bounding boxes, which must be converted to token-level labels before applying the token-area metric.

## Evidence (verbatim from paper)

> As the inputs of our model are serialized 2-D documents, the typical BIO-tagging evaluation is not suitable for our task. The tokens of each semantic unit may discontinuously distribute in the input sequence. In this case, we proposed a new metric, especially for text-based document layout analysis methods. For each kind of document semantic structure, we calculated their metrics individually. The definition is as follows: Precision = Area of Ground truth tokens in Detected tokens / Area of all Detected tokens, Recall = Area of Ground truth tokens in Detected tokens / Area of all Ground truth tokens, F1 Score = 2 * Precision * Recall / (Precision + Recall).

## Citation

```bibtex
@misc{li2020docbank,
  title={DocBank: A Benchmark Dataset for Document Layout Analysis},
  author={Minghao Li et al. (2020)},
  year={2020},
  note={arXiv:2006.01038}
}
```

- arXiv: 2006.01038