Gistbench Eval

---
name: gistbench-eval
description: Evaluates LLMs' ability to extract and verify user interests from interaction histories, focusing on factual grounding, specificity, and strict instruction following across heterogeneous engagement types. Use when the user wants to benchmark on Unspecified real-world engagement datasets, or asks about evaluating this task. Reports IG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29112
  bibtex_key: fostiropoulos2026gistbench
  confidence: high
---

# gistbench-eval

> GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification — Fostiropoulos et al. (2026) (arXiv:2603.29112, 2026)

## What this evaluates

Evaluates LLMs' ability to extract and verify user interests from interaction histories, focusing on factual grounding, specificity, and strict instruction following across heterogeneous engagement types.

## Datasets

- **Unspecified real-world engagement datasets** — total ?; splits: (unstated); repo https://github.com/facebookresearch/GISTBench

## Metrics

- `IG` **(primary)** — range: [0, 1]
  - Decomposed into precision and recall to penalize hallucinations and reward coverage of verified interests.
- `IS` — range: [0, 1]
  - Measures the distinctiveness of verified interests.

## Input / output format

**Input**: Chunked user interaction history (UIH) with up to 100 engagements per prompt, formatted as numbered entries containing interaction type and object description, along with dataset-specific grounding thresholds in the system prompt.

**Output**: Model-generated list of identified interests with evidence citations referencing sequential object identifiers, ideally structured as JSON.

## Scoring recipe

```python
def compute_metrics(model_output, uih_chunks):
    pred = json_extract(repair(model_output))
    if pred is None: return None
    ig_precision = judge_l33_70b.verify_precision(pred.interests, uih_chunks)
    ig_recall = judge_l33_70b.verify_recall(pred.interests, uih_chunks)
    ig = harmonic_mean(ig_precision, ig_recall)
    is_score = judge_l33_70b.compute_distinctiveness(pred.interests)
    return {"IG": ig, "IS": is_score}
```

## Common pitfalls

- JSON parsing failures require multi-stage extraction and repair before scoring.
- UIH chunking requires aggregation at the user level to avoid losing context across windows.
- Unparsable generations are strictly filtered out, which may bias results if a model consistently fails formatting.

## Evidence (verbatim from paper)

> The IG evidence-filtering judge and IS retrieval judge both used Llama-3.3-70B-Instruct. All intermediate outputs undergo best-effort JSON extraction: we first attempt standard parsing, then apply regex-based extraction to recover JSON objects embedded in free-text responses, and finally use automated JSON repair to fix common structural errors (unclosed brackets, trailing commas). Generations that remain unparsable after these recovery steps are filtered before metric computation.

## Citation

```bibtex
@misc{fostiropoulos2026gistbench,
  title={GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification},
  author={Fostiropoulos et al. (2026)},
  year={2026},
  note={arXiv:2603.29112}
}
```

- arXiv: 2603.29112