Extractbench Eval

---
name: extractbench-eval
description: Evaluates LLMs on complex, schema-driven PDF-to-JSON structured extraction, testing their ability to handle nested objects, arrays, heterogeneous field types, and strict correctness criteria across enterprise-scale documents. Use when the user wants to benchmark on ExtractBench, or asks about evaluating this task. Reports Pass Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12247
  bibtex_key: ferguson2026extractbench
  confidence: high
---

# extractbench-eval

> ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — Ferguson et al. (2026) (arXiv:2602.12247, 2026)

## What this evaluates

Evaluates LLMs on complex, schema-driven PDF-to-JSON structured extraction, testing their ability to handle nested objects, arrays, heterogeneous field types, and strict correctness criteria across enterprise-scale documents.

## Datasets

- **ExtractBench** — total 35; splits: benchmark (35); repo https://github.com/ContextualAI/extract-bench

## Metrics

- `Pass Rate` **(primary)** — range: percent
  - Field-level pass rate where invalid extractions contribute 0. Computed as the fraction of fields that pass their schema-defined metric.
- `Valid JSON` — range: percent
  - Fraction of model outputs that are parseable and conform to the provided JSON Schema.
- `Acc (Valid)` — range: percent
  - Pass rate conditioned only on extractions that produced valid JSON.
- `string_semantic` — range: [0, 1]
  - LLM-based equivalence metric for free text, allowing domain-specific instructions for acceptable variations.
- `number_tolerance` — range: [0, 1]
  - Numeric comparison within a specified margin (e.g., 0.1%).
- `array_llm` — range: percent
  - Semantic alignment of object arrays using an LLM judge to map matched pairs, missed gold items, and spurious predictions, then computing precision, recall, and F1.

## Input / output format

**Input**: PDF documents paired with a JSON Schema that defines the target extraction structure and per-field evaluation configurations.

**Output**: A JSON object containing extracted fields matching the schema structure. Must be parseable and schema-conforming to be considered valid.

## Scoring recipe

```python
def evaluate(gold, pred, schema):
    if not is_valid_json(pred, schema):
        return {"valid_json": False, "pass_rate": 0.0}
    scores = []
    for field in schema.fields:
        g_val = gold.get(field.name)
        p_val = pred.get(field.name)
        if g_val is None and p_val is None:
            scores.append(True)
        elif g_val is None and p_val is not None:
            scores.append(False) # hallucination
        elif g_val is not None and p_val is None:
            scores.append(False) # omission
        else:
            scores.append(field.metric(g_val, p_val, field.config))
    return {"valid_json": True, "pass_rate": sum(scores)/len(scores)}
```

## Common pitfalls

- Applying a single global metric (e.g., exact match) across all fields ignores heterogeneous correctness criteria like numeric tolerance or semantic equivalence.
- Ignoring the three-way missing value distinction (present, null, MISSING) conflates omissions with hallucinations and structural errors.
- Using position-based matching for arrays fails when models reorder, omit, or duplicate items, leading to artificially low scores.

## Evidence (verbatim from paper)

> Pass Rate: field-level pass rate (invalid extractions contribute 0). Acc (Valid): pass rate conditioned on valid extraction only.

## Citation

```bibtex
@misc{ferguson2026extractbench,
  title={ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction},
  author={Ferguson et al. (2026)},
  year={2026},
  note={arXiv:2602.12247}
}
```

- arXiv: 2602.12247