Jsonschemabench Coverage Eval

---
name: jsonschemabench-coverage-eval
description: Evaluates whether language models can generate JSON objects that strictly comply with a given JSON Schema under constrained decoding. It measures the empirical coverage of schema features supported by the model and decoding framework. Use when the user wants to benchmark on JSONSchemaBench, or asks about evaluating this task. Reports Top 1 Empirical Coverage.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.10868
  bibtex_key: [REDACTED]
  confidence: high
---

# jsonschemabench-coverage-eval

> JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models — Geng et al. (2025) (arXiv:2501.10868, 2025)

## What this evaluates

Evaluates whether language models can generate JSON objects that strictly comply with a given JSON Schema under constrained decoding. It measures the empirical coverage of schema features supported by the model and decoding framework.

## Datasets

- **JSONSchemaBench** — total 10000; splits: test (10000); repo https://github.com/guidance-ai/jsonschemabench

## Metrics

- `Top 1 Empirical Coverage` **(primary)** — range: percent
  - The proportion of schemas for which the single top-1 generated JSON object passes validation against the provided JSON Schema (Draft-2020-12 with format checks enabled).

## Input / output format

**Input**: A JSON Schema (Draft-2020-12) embedded in a prompt template.

**Output**: A single JSON object generated via greedy decoding.

## Scoring recipe

```python
compliant = 0
for schema, gen_json in zip(schemas, outputs):
    try:
        jsonschema.validate(gen_json, schema, format_checker=jsonschema.FormatChecker())
        compliant += 1
    except jsonschema.ValidationError:
        pass
coverage = compliant / len(schemas)
```

## Common pitfalls

- Only generating a single top-1 output may underestimate true model coverage, as sampling multiple outputs could reveal compliance on harder schemas.
- The `jsonschema` library does not guarantee validation of all schema constraints, even with format checks enabled, potentially leading to a slight overestimation of compliance.

## Evidence (verbatim from paper)

> We use greedy decoding with no top P or top K sampling for all the experiments. We only get one output from the model, which we will use to validate the schema compliance. It’s totally plausible to sample more outputs and validate them all, and it might detect more schema violations. The fact that we only sample the top 1 output may quantify our empirical coverage as Top 1 Empirical Coverage. We use the jsonschema library with the Draft-2020-12 version of the JSON Schema standard to validate the generated JSON object. We turn on the ‘format’ checks, which are not enabled by default in Python. Strictly speaking, the jsonschema library doesn’t guarantee the validation of all the schema constraints, even with the ‘format’ checks enabled. It is possible, though very rare, for a schema-noncompliant output to be validated as compliant by the jsonschema library, leading to a slight overestimation of empirical coverage. However, such occurrences are corner cases and happen infrequently.

## Citation

```bibtex
@misc{geng2025jsonschemabench,
  title={JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2501.10868}
}
```

- arXiv: 2501.10868