Horizonbench Eval

---
name: horizonbench-eval
description: Probes long-horizon personalization and belief-update capability. It tests whether models can track evolving user preferences across ~6 months of conversation history and correctly select responses aligned with updated preferences, rather than anchoring on outdated values. Use when the user wants to benchmark on HorizonBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17283
  bibtex_key: li2026horizonbench
  confidence: high
---

# horizonbench-eval

> HorizonBench: Long-Horizon Personalization with Evolving Preferences — Li et al. (2026) (arXiv:2604.17283, 2026)

## What this evaluates

Probes long-horizon personalization and belief-update capability. It tests whether models can track evolving user preferences across ~6 months of conversation history and correctly select responses aligned with updated preferences, rather than anchoring on outdated values.

## Datasets

- **HorizonBench** — total 4245; splits: test (4245); repo https://github.com/stellalisy/HorizonBench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of items where the model correctly selects the gold assistant response out of five options.
- `pre-evolution distractor selection rate` — range: [0, 1]
  - Proportion of wrong answers that select the option reflecting the user's original, outdated preference.

## Input / output format

**Input**: Full conversation history (~6 months, ~4,300 turns, ~163K tokens) and five candidate assistant responses.

**Output**: Selection of exactly one of the five candidate responses.

## Scoring recipe

```python
correct = 0
distractor_wrong = 0
total_wrong = 0
for item in dataset:
    pred = model.generate(item.history, item.options)
    if pred == item.gold:
        correct += 1
    else:
        total_wrong += 1
        if pred == item.pre_evolution_distractor:
            distractor_wrong += 1
accuracy = correct / len(dataset)
distractor_rate = distractor_wrong / total_wrong if total_wrong > 0 else 0
```

## Common pitfalls

- Truncating long histories by removing earlier turns can artificially lower performance or alter preference tracking, so results are reported on history-filtered items only.
- The 20% chance baseline applies to the 5-option task, but models often score below it due to systematic anchoring on the pre-evolution distractor rather than uniform random guessing.
- Position bias is minor (<1 pp difference from raw accuracy) and should not be cited as the primary explanation for performance gaps.

## Evidence (verbatim from paper)

> We report results on the combined multi-generator dataset (4,245 items from 360 users across three generators). Items exceeding a model’s context window are truncated by removing earlier conversation turns. Figure 3 reports per-model accuracy on the 4,245 history-filtered items.

## Citation

```bibtex
@misc{li2026horizonbench,
  title={HorizonBench: Long-Horizon Personalization with Evolving Preferences},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.17283}
}
```

- arXiv: 2604.17283