Doc Key Info Extraction Eval

---
name: doc-key-info-extraction-eval
description: Evaluates a model's ability to extract fine-grained key information categories (e.g., total price, date, address) from unstructured, template-agnostic document images. It probes robustness to spatial layout variations, dual-modality feature fusion, and generalization to unseen templates and OCR noise. Use when the user wants to benchmark on SROIE, WildReceipt, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.14470
  bibtex_key: sun2021spatial
  confidence: high
---

# doc-key-info-extraction-eval

> Spatial Dual-Modality Graph Reasoning for Key Information Extraction — Hongbin Sun et al. (arXiv:2103.14470, 2021)

## What this evaluates

Evaluates a model's ability to extract fine-grained key information categories (e.g., total price, date, address) from unstructured, template-agnostic document images. It probes robustness to spatial layout variations, dual-modality feature fusion, and generalization to unseen templates and OCR noise.

## Datasets

- **SROIE** — total ?; splits: (unstated)
- **WildReceipt** — total ?; splits: (unstated)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall for key information category prediction. Calculated per category and averaged across 12 value categories on WildReceipt, or reported as a single overall score on SROIE. Values are expressed as percentages.

## Input / output format

**Input**: Document images resized to 512x512, accompanied by text bounding boxes and recognized text strings (either ground truth or OCR-generated). Visual features are extracted via U-Net, and textual features via Bi-LSTM.

**Output**: Predicted key information category label for each text region/node in the document graph (e.g., 'Total', 'Tax', 'Date', 'Prod item').

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    fp = sum(1 for p, g in zip(predictions, gold) if p != g)
    fn = sum(1 for p, g in zip(predictions, gold) if p != g)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Performance drops significantly (~6.3%) when using OCR-recognized text instead of ground truth text due to character misrecognition.
- Mismatch between detected and ground truth text boxes (e.g., 1-to-many overlaps) introduces noisy signals if matched by maximum IoU alone.
- Ablation shows removing graph reasoning causes an 11.5% F1 drop, highlighting that spatial message passing is critical for this task.

## Evidence (verbatim from paper)

> Specifically, SDMG-R achieves 11.8%, 9.7%, and 3.0% absolute improvements in terms of $F_{1}$ score averaged on 12 value categories on WildReceipt compared with Chargrid, Chargrid-UNet, and VRD respectively.

## Citation

```bibtex
@misc{sun2021spatial,
  title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
  author={Hongbin Sun et al.},
  year={2021},
  note={arXiv:2103.14470}
}
```

- arXiv: 2103.14470