Visual Information Extraction Eval

---
name: visual-information-extraction-eval
description: Evaluates a model's ability to extract entity spans and link them to key-value pairs from complex, real-world document images. It probes joint vision-language understanding, handling poor image quality, occlusion, and multi-lingual text without relying on external OCR pipelines. Use when the user wants to benchmark on FUNSD, XFUND, CORD, SIBR, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.13095
  bibtex_key: [REDACTED]
  confidence: high
---

# visual-information-extraction-eval

> Modeling Entities as Semantic Points for Visual Information Extraction in the Wild — Yang et al. (2023) (arXiv:2303.13095, 2023)

## What this evaluates

Evaluates a model's ability to extract entity spans and link them to key-value pairs from complex, real-world document images. It probes joint vision-language understanding, handling poor image quality, occlusion, and multi-lingual text without relying on external OCR pipelines.

## Datasets

- **FUNSD** — total 199; splits: train (149), test (50)
- **XFUND** — total ?; splits: train (-1), test (-1)
- **CORD** — total 1000; splits: train (800), val (100), test (100)
- **SIBR** — total 1000; splits: test (1000)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Harmonic mean of precision and recall for Entity Extraction (EE) and Entity Linking (EL). Calculated as 2 * (precision * recall) / (precision + recall). Precision and recall are computed by matching predicted boxes, categories, and links against ground truth annotations.

## Input / output format

**Input**: Document images. Depending on the evaluation setting, inputs may include ground-truth or predicted text bounding boxes and OCR text contents.

**Output**: Predicted bounding boxes for entities, their categorical labels (for EE), and key-value pair links (for EL).

## Scoring recipe

```python
# For FUNSD/XFUND/CORD: aggregate multi-segment predictions into single entities
preds = aggregate_segments(predictions)
gold = aggregate_segments(gold)
# Match predicted boxes/labels/links to ground truth
tp_ee, fp_ee, fn_ee = match_entities(preds, gold, task="EE")
tp_el, fp_el, fn_el = match_links(preds, gold, task="EL")
# Compute F1 for each task
f1_ee = 2 * tp_ee / (2 * tp_ee + fp_ee + fn_ee)
f1_el = 2 * tp_el / (2 * tp_el + fp_el + fn_el)
return f1_ee, f1_el
```

## Common pitfalls

- Comparing against LayoutLMv3 is unfair because it uses ground-truth text contents during evaluation, while the proposed method does not.
- On SIBR, the evaluation is strictly end-to-end; using external OCR or ground-truth annotations during inference will invalidate the reported F1 scores.
- Entity segments spanning multiple text lines must be aggregated into a single entity before matching to avoid undercounting.

## Evidence (verbatim from paper)

> Following [19], we aggregate segment features of the text into one entity if the entity has multiple segments on FUNSD, XFUND, and CORD. But when testing end-to-end on SIBR, model inference does not rely on any annotation information. After decoding, the boxes, categories, and links are directly matched with GT. We use F1-score as our evaluation metrics for both EE and EL tasks.

## Citation

```bibtex
@misc{yang2023modeling,
  title={Modeling Entities as Semantic Points for Visual Information Extraction in the Wild},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2303.13095}
}
```

- arXiv: 2303.13095