Horizon Eval

---
name: horizon-eval
description: Evaluates user behavior modeling capabilities across temporal generalization, cross-domain prediction, and unseen-user scenarios. It probes how well recommendation models and LLMs can generalize to out-of-distribution users and future time periods using real-world interaction sequences. Use when the user wants to benchmark on Amazon Reviews (HORIZON Benchmark), or asks about evaluating this task. Reports NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17259
  bibtex_key: goel2026horizon
  confidence: high
---

# horizon-eval

> HORIZON: A Benchmark for In-the-wild User Behaviour Modeling — Goel et al. (2026) (arXiv:2604.17259, 2026)

## What this evaluates

Evaluates user behavior modeling capabilities across temporal generalization, cross-domain prediction, and unseen-user scenarios. It probes how well recommendation models and LLMs can generalize to out-of-distribution users and future time periods using real-world interaction sequences.

## Datasets

- **Amazon Reviews (HORIZON Benchmark)** — total 54000000; splits: IND pool (53000000), OOD set (1000000), Training subset (100000), IND test (25000), OOD test (25000)

## Metrics

- `NDCG@K` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank K. Sum of relevance scores discounted by log2(rank+1), normalized by ideal DCG.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank. Average of 1/rank of the first relevant item across queries.
- `Recall@K` — range: [0, 1]
  - Fraction of relevant items retrieved in the top K recommendations.
- `Precision@K` — range: [0, 1]
  - Fraction of retrieved items in the top K that are relevant.

## Input / output format

**Input**: User interaction history sequences (item IDs with timestamps) for recommendation tasks; for LLM tasks, standardized prompts encoding user behavior and item descriptions/IDs.

**Output**: Ranked list of recommended item IDs (top K) for recommendation tasks; retrieved item IDs for LLM retrieval tasks.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k_values=[10, 50, 100]):
    scores = {}
    for k in k_values:
        top_k = predictions[:k]
        scores[f'Recall@{k}'] = 1 if any(item in ground_truth for item in top_k) else 0
        scores[f'Precision@{k}'] = len(set(top_k) & set(ground_truth)) / k
        dcg = sum(1 / math.log2(i + 2) for i, item in enumerate(top_k) if item in ground_truth)
        idcg = sum(1 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
        scores[f'NDCG@{k}'] = dcg / idcg if idcg > 0 else 0
    for i, item in enumerate(predictions):
        if item in ground_truth:
            scores['MRR'] = 1 / (i + 1)
            break
    return scores
```

## Common pitfalls

- Temporal cutoff τ=2020 is strictly enforced; using standard random or leave-one-out splits violates the benchmark's temporal generalization protocol.
- OOD and IND test sets are disjoint; mixing them or using the same pool for fine-tuning and evaluation causes data leakage and invalidates zero-shot/fine-tuning comparisons.
- Metrics are reported at K=10, 50, 100; reporting only K=10 or using ratio-based/leave-one-out evaluation strategies yields non-comparable results.

## Evidence (verbatim from paper)

> All models are trained with standardized hyperparameters and evaluated on our four evaluation settings using MRR, Recall@K, and NDCG@K for K=10,50,100. As we do not perform ranking across queries, we compute standard retrieval metrics i.e. Recall@K and Precision@K for K=10,50,100 to assess the effectiveness of the generated outputs in retrieving relevant items.

## Citation

```bibtex
@misc{goel2026horizon,
  title={HORIZON: A Benchmark for In-the-wild User Behaviour Modeling},
  author={Goel et al. (2026)},
  year={2026},
  note={arXiv:2604.17259}
}
```

- arXiv: 2604.17259