Sbr Intent Eval

---
name: sbr-intent-eval
description: Evaluates a session-based recommendation model's ability to predict the next item in a user session using validated and enriched LLM-generated intents. It probes the model's capacity to leverage semantic intent signals alongside sequential interaction patterns for accurate item ranking. Use when the user wants to benchmark on Beauty (Amazon), Yelp, Books (Amazon), or asks about evaluating this task. Reports Hit Rate@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.00570
  bibtex_key: [REDACTED]
  confidence: high
---

# sbr-intent-eval

> Session-Based Recommendation with Validated and Enriched LLM Intents — Lee et al. (2025) (arXiv:2508.00570, 2025)

## What this evaluates

Evaluates a session-based recommendation model's ability to predict the next item in a user session using validated and enriched LLM-generated intents. It probes the model's capacity to leverage semantic intent signals alongside sequential interaction patterns for accurate item ranking.

## Datasets

- **Beauty (Amazon)** — total 7524; splits: train (6448), val (611), test (465)
- **Yelp** — total 41892; splits: train (34296), val (3985), test (3611)
- **Books (Amazon)** — total 20102; splits: train (16864), val (1723), test (1515)

## Metrics

- `Hit Rate@10` **(primary)** — range: [0, 1]
  - Fraction of test sessions where the ground-truth next item appears in the top-10 predicted items. Computed as the mean over all test sessions.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 10. Ranks the ground-truth item by its position in the predicted list, applying a logarithmic discount. Averaged over all test sessions.

## Input / output format

**Input**: A sequence of item IDs representing a user's session history, optionally augmented with item metadata or LLM-derived intent embeddings.

**Output**: A ranked list of candidate items (typically top-20 for evaluation), from which the ground-truth next item's rank is extracted.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k=10):
    hits = 0
    ndcg_sum = 0.0
    for pred_list, true_item in zip(predictions, ground_truth):
        rank = pred_list.index(true_item) + 1 if true_item in pred_list else k + 1
        if rank <= k:
            hits += 1
            ndcg_sum += 1.0 / math.log2(1 + rank)
    hr = hits / len(ground_truth)
    ndcg = ndcg_sum / len(ground_truth)
    return hr, ndcg
```

## Common pitfalls

- Splitting data by individual interaction timestamp instead of session timestamp causes severe data leakage in session-based recommendation.
- Averaging metrics over items instead of sessions, or vice versa, changes the evaluation scale significantly.
- Evaluating on top-5 or top-20 without specifying the cutoff leads to incomparable results.

## Evidence (verbatim from paper)

> We evaluate recommendation quality using Hit Rate (H) and NDCG (N) at cutoffs {5, 10, 20}, following prior studies*(Liu et al., [2024b]; Sun et al., [2024]; Ren et al., [2024]; Liu et al., [2023], [2024a])*. All results are averaged over five independent runs with different random seeds.

## Citation

```bibtex
@misc{lee2025sessionbasedrecommendation,
  title={Session-Based Recommendation with Validated and Enriched LLM Intents},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2508.00570}
}
```

- arXiv: 2508.00570