Docile Eval

---
name: docile-eval
description: Evaluates a model's ability to localize and extract key information (KILE) and recognize line items (LIR) in business documents. It probes multi-modal document understanding, specifically token-level classification and bounding box merging based on OCR and layout features. Use when the user wants to benchmark on DocILE, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.05658
  bibtex_key: simsa2023docile
  confidence: high
---

# docile-eval

> DocILE Benchmark for Document Information Localization and Extraction — Štepan Šimsa et al. (2023) (arXiv:2302.05658, 2023)

## What this evaluates

Evaluates a model's ability to localize and extract key information (KILE) and recognize line items (LIR) in business documents. It probes multi-modal document understanding, specifically token-level classification and bounding box merging based on OCR and layout features.

## Datasets

- **DocILE** — total 106700; splits: test (-1); repo https://github.com/rossumai/docile

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall computed over all predicted token-class assignments using a standard BIO tagging scheme. Matches are determined by overlapping predicted and ground-truth bounding boxes or token sequences.
- `AP` — range: [0, 1]
  - Average Precision computed from the precision-recall curve across different confidence thresholds for the multi-label classification task.
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive token-class assignments to the total number of predicted positive assignments.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive token-class assignments to the total number of actual positive assignments in the ground truth.

## Input / output format

**Input**: Document images paired with OCR-extracted tokens, including their bounding boxes and text content. Tokens are pre-ordered in top-down, left-to-right reading order per text line.

**Output**: For each OCR token, a predicted class label from the 55 fine-grained KILE/LIR classes using a BIO tagging scheme, plus line-item boundary tags (B-I, I-I, O-I, E-I). Predictions are merged horizontally/vertically based on spatial proximity thresholds to form final text blocks and bounding boxes.

## Scoring recipe

```python
preds = model.predict(doc_image, ocr_tokens)
gold = doc.gold_annotations

true_pos = 0
false_pos = 0
false_neg = 0

for pred_group in preds:
    matched = False
    for gold_group in gold:
        if match(pred_group, gold_group, threshold=0.5):
            true_pos += 1
            matched = True
            break
    if not matched:
        false_pos += 1

for gold_group in gold:
    if not any(match(pred_group, gold_group) for pred_group in preds):
        false_neg += 1

precision = true_pos / (true_pos + false_pos) if (true_pos + false_pos) > 0 else 0
recall = true_pos / (true_pos + false_neg) if (true_pos + false_neg) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- OCR tokens must be reordered in top-down, left-to-right order before processing; skipping this degrades performance significantly.
- Pre-training on external document datasets (e.g., IIT-CDIP) is strictly prohibited in the official benchmark, though some baselines may violate this.
- The merging strategy for final predictions is simplistic and relies on fixed spatial thresholds (e.g., 25% margin on text block height), which may not generalize to all layouts.

## Evidence (verbatim from paper)

> The primary metric for each task is shown in bold. Table 3: Baseline results for KILE & LIR. LayoutLMv3BASE, achieving the best results, was pre-trained on another document dataset - IIT-CDIP [37], which is prohibited in the official benchmark. The best results among permitted models are underlined. The primary metric for each task is shown in bold.
<table><tr><td rowspan="2">Model</td><td colspan="4">KILE</td><td colspan="4">LIR</td></tr><tr><td>F1</td><td>AP</td><td>Prec.</td><td>Recall</td><td>F1</td><td>AP</td><td>Prec.</td><td>Recall</td></tr>

## Citation

```bibtex
@misc{simsa2023docile,
  title={DocILE Benchmark for Document Information Localization and Extraction},
  author={Štepan Šimsa et al. (2023)},
  year={2023},
  note={arXiv:2302.05658}
}
```

- arXiv: 2302.05658