Structured Output Benchmark Eval

---
name: structured-output-benchmark-eval
description: Evaluates large language models' ability to extract structured information from multi-modal sources (text, images, audio) into valid JSON formats, isolating schema compliance from value accuracy. Use when the user wants to benchmark on Multi-Source Structured Output Benchmark, or asks about evaluating this task. Reports correct_value_extraction.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.25359
  bibtex_key: singh2026sob
  confidence: medium
---

# structured-output-benchmark-eval

> The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models — Singh et al. (2026) (arXiv:2604.25359, 2026)

## What this evaluates

Evaluates large language models' ability to extract structured information from multi-modal sources (text, images, audio) into valid JSON formats, isolating schema compliance from value accuracy.

## Datasets

- **Multi-Source Structured Output Benchmark** — total ?; splits: test (-1)

## Metrics

- `correct_value_extraction` **(primary)** — range: [0, 1]
  - Measures the correctness of extracted values against ground truth across text, image, and audio sources, independent of schema validity.

## Input / output format

**Input**: Context (text/image/audio), question, and a JSON schema defining the expected output structure.

**Output**: A conforming JSON response containing the extracted structured data.

## Scoring recipe

```python
def score(predictions, gold, schema):
    schema_valid = validate_json_schema(predictions, schema)
    value_correct = exact_match(predictions, gold)
    return {
        'schema_compliance': schema_valid,
        'correct_value_extraction': value_correct,
        'overall_fidelity': schema_valid and value_correct
    }
```

## Common pitfalls

- Hallucinations in structured fields are harder to detect because syntactically correct JSON can still contain incorrect values.
- Model size does not correlate with extraction performance, contradicting typical scaling expectations.
- Evaluating in reasoning mode conflates extraction capability with compute budget and problem-solving ability.

## Evidence (verbatim from paper)

> It reveals a stark gap between schema validity and correct value extraction—83.0% on text, 67.2% on images, 23.7% on audio—showing that model size does not correlate with performance and that hallucinations in structured fields are harder to detect due to syntactic correctness.

## Citation

```bibtex
@misc{singh2026sob,
  title={The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models},
  author={Singh et al. (2026)},
  year={2026},
  note={arXiv:2604.25359}
}
```

- arXiv: 2604.25359