Onerec Think Eval

---
name: onerec-think-eval
description: Evaluates a generative recommendation model's ability to perform sequential next-item prediction and in-text reasoning for user preference alignment. It probes the model's capacity to generate interpretable reasoning paths alongside item recommendations, and measures ranking accuracy on standard recommendation benchmarks. Use when the user wants to benchmark on Amazon Beauty, Amazon Toys, Amazon Sports, or asks about evaluating this task. Reports R@K / NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.11639
  bibtex_key: liu2025onerecthink
  confidence: high
---

# onerec-think-eval

> OneRec-Think: In-Text Reasoning for Generative Recommendation — Liu et al. (2025) (arXiv:2510.11639, 2025)

## What this evaluates

Evaluates a generative recommendation model's ability to perform sequential next-item prediction and in-text reasoning for user preference alignment. It probes the model's capacity to generate interpretable reasoning paths alongside item recommendations, and measures ranking accuracy on standard recommendation benchmarks.

## Datasets

- **Amazon Beauty** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon Toys** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon Sports** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `R@K / NDCG@K` **(primary)** — range: [0, 1]
  - Recall@K: 1 if the ground-truth item appears in the top-K predicted items, else 0. NDCG@K: Normalized Discounted Cumulative Gain at K, which weights the relevance of the ground-truth item by the logarithmic position in the ranked list and normalizes by the ideal DCG.

## Input / output format

**Input**: Sequential user interaction history (item tokens) for next-item prediction.

**Output**: Generated sequence of item tokens, optionally accompanied by in-text reasoning steps.

## Scoring recipe

```python
def compute_metrics(preds, gold, k=5):
    recall = 1.0 if gold in preds[:k] else 0.0
    if gold in preds:
        rank = preds.index(gold) + 1
        dcg = 1.0 / math.log2(rank + 1)
        idcg = 1.0 / math.log2(2)
        ndcg = dcg / idcg
    else:
        ndcg = 0.0
    return recall, ndcg
```

## Common pitfalls

- Exact dataset split sizes are not provided in the main text; only the Amazon review benchmark source is cited.
- K is fixed at 5 and 10; results are not reported for other cutoffs.
- Industrial A/B test metrics (e.g., App Stay Time) are evaluated separately from the academic benchmark metrics and should not be conflated.

## Evidence (verbatim from paper)

> We use three real-world recommendation datasets from the popular Amazon review benchmark: Beauty, Toys, and Sports. We compare OneRec-Think against two groups of competitive baselines: (1) Classic sequential methods like BERT4Rec, HGN, GRU4Rec, and SASRec; and (2) Generative Recommender Models, such as TIGER, HSTU, and ReaRec. Top-K Recall (R@K) and NDCG (N@K) with K=5 and 10 are used as metrics, following Rajput et al. ([2023]).

## Citation

```bibtex
@misc{liu2025onerecthink,
  title={OneRec-Think: In-Text Reasoning for Generative Recommendation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.11639}
}
```

- arXiv: 2510.11639