Construct Eval

---
name: construct-eval
description: Evaluates the trustworthiness and accuracy of LLM-generated structured outputs (JSON) against a ground truth or expected schema. It probes the model's ability to detect per-field and per-document errors in data extraction tasks without requiring labeled data. Use when the user wants to benchmark on Four real-world datasets (unspecified in excerpt), or asks about evaluating this task. Reports rating.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.18014
  bibtex_key: [REDACTED]
  confidence: high
---

# construct-eval

> Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction — Goh et al. (2026) (arXiv:2603.18014, 2026)

## What this evaluates

Evaluates the trustworthiness and accuracy of LLM-generated structured outputs (JSON) against a ground truth or expected schema. It probes the model's ability to detect per-field and per-document errors in data extraction tasks without requiring labeled data.

## Datasets

- **Four real-world datasets (unspecified in excerpt)** — total ?; splits: test (-1); repo https://github.com/cleanlab/structured-output-benchmark

## Metrics

- `rating` **(primary)** — range: [1, 10]
  - An LLM judge evaluates the generator's structured output against the user request and schema, outputting a score from 1 to 10. Higher scores indicate greater trustworthiness and accuracy.

## Input / output format

**Input**: User request (including document to process), desired structured output schema, and the generator model's structured output response.

**Output**: Per-document: a short explanation followed by a rating in format 'Rating: [[x]]' (1-10). Per-field: a JSON object mapping each field name to an object containing 'explanation' (str) and 'rating' (int, 1-10).

## Scoring recipe

```python
def score_trustworthiness(input_text, schema, generated_output):
    # Per-document scoring
    doc_prompt = build_prompt(input_text, schema, generated_output)
    doc_response = call_llm_judge(doc_prompt)
    doc_score = extract_int(doc_response, pattern=r'\[\[(\d+)\]\]')
    
    # Per-field scoring
    field_scores = {}
    for field_name in schema.keys():
        field_prompt = build_prompt(input_text, schema, generated_output, ignore_others=[field_name])
        field_response = call_llm_judge(field_prompt)
        field_scores[field_name] = extract_int(field_response, pattern=r'\[\[(\d+)\]\]')
    return doc_score, field_scores
```

## Common pitfalls

- The LLM-as-Judge prompt is adapted from Zheng et al. (2023) and may inherit calibration biases or positional biases common to LLM judges.
- Per-field scoring requires precise schema parsing to map judge outputs back to original fields, especially for nested or dynamically generated JSON structures.
- Ratings are subjective 1-10 scores and may not linearly correlate with exact ground-truth accuracy without thresholding or calibration.

## Evidence (verbatim from paper)

> After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[rating]", for example: "Rating: [[5]]".

## Citation

```bibtex
@misc{goh2026construct,
  title={Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction},
  author={Goh et al. (2026)},
  year={2026},
  note={arXiv:2603.18014}
}
```

- arXiv: 2603.18014