Llmstructbench Eval

---
name: llmstructbench-eval
description: Evaluates large language models' ability to extract structured data from natural-language emails into valid JSON objects that adhere to a provided schema. It measures both syntactic validity (structural correctness) and semantic accuracy (correct value extraction) across varying levels of JSON nesting complexity. Use when the user wants to benchmark on LLMStructBench, or asks about evaluating this task. Reports DOC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.14743
  bibtex_key: tenckhoff2026llmstructbench
  confidence: high
---

# llmstructbench-eval

> LLMStructBench: Benchmarking Large Language Model Structured Data Extraction — Tenckhoff et al. (2026) (arXiv:2602.14743, 2026)

## What this evaluates

Evaluates large language models' ability to extract structured data from natural-language emails into valid JSON objects that adhere to a provided schema. It measures both syntactic validity (structural correctness) and semantic accuracy (correct value extraction) across varying levels of JSON nesting complexity.

## Datasets

- **LLMStructBench** — total 995; splits: test (995)

## Metrics

- `DOC` **(primary)** — range: [0, 1]
  - DOC measures document-level structural compliance, evaluating whether the generated JSON strictly adheres to the provided schema and is syntactically valid.
- `MK` — range: [0, 1]
  - MK evaluates key-level matching accuracy, measuring the proportion of correctly extracted keys relative to the ground truth.
- `MV` — range: [0, 1]
  - MV measures value-level matching accuracy, assessing the correctness of extracted values against the ground truth.
- `WV` — range: [0, 1]
  - WV computes a weighted value match score, accounting for the importance or frequency of specific fields in the extraction task.

## Input / output format

**Input**: Natural-language message, a representative example input-output pair, and a JSON schema defining required keys, data types, and nesting depth.

**Output**: A single JSON object adhering strictly to the provided schema.

## Scoring recipe

```python
def evaluate(gen_output, gt_json, schema):
    try:
        parsed = json.loads(gen_output)
        is_valid_schema = validate_schema(parsed, schema)
    except json.JSONDecodeError:
        return {'DOC': 0.0, 'MK': 0.0, 'MV': 0.0, 'WV': 0.0}
    mk = compute_key_match(parsed, gt_json)
    mv = compute_value_match(parsed, gt_json)
    wv = compute_weighted_value_match(parsed, gt_json)
    return {'DOC': float(is_valid_schema), 'MK': float(mk), 'MV': float(mv), 'WV': float(wv)}
```

## Common pitfalls

- Confusing syntactic validity (schema compliance/valid JSON) with semantic accuracy (correct extraction of values from text).
- Assuming model size is the main performance driver; the benchmark shows prompting strategy (e.g., schema-driven or two-step) has a greater impact on parsing reliability.
- Treating the dataset as purely real-world; it is synthetically generated from structured JSON via GPT-4o and manually verified, which may not capture all real-world email noise.

## Evidence (verbatim from paper)

> The benchmark introduces novel metrics (e.g., DOC, MK, MV, WV) that separate syntactic validity from semantic accuracy, enabling fine-grained analysis of model and prompting performance in end-to-end information extraction workflows.

## Citation

```bibtex
@misc{tenckhoff2026llmstructbench,
  title={LLMStructBench: Benchmarking Large Language Model Structured Data Extraction},
  author={Tenckhoff et al. (2026)},
  year={2026},
  note={arXiv:2602.14743}
}
```

- arXiv: 2602.14743