Lru Rec Eval

---
name: lru-rec-eval
description: Evaluates sequential recommendation models on next-item prediction tasks across various domains (movies, products, games) with varying sequence lengths and sparsity. Use when the user wants to benchmark on ML-1M, Amazon-Beauty, Amazon-Video, Amazon-Sports, Steam, XLong, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.02367
  bibtex_key: yue2023lru
  confidence: high
---

# lru-rec-eval

> Linear Recurrent Units for Sequential Recommendation — Yue et al. (2023) (arXiv:2310.02367, 2023)

## What this evaluates

Evaluates sequential recommendation models on next-item prediction tasks across various domains (movies, products, games) with varying sequence lengths and sparsity.

## Datasets

- **ML-1M** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Beauty** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Video** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Sports** — total ?; splits: train (-1), val (-1), test (-1)
- **Steam** — total ?; splits: train (-1), val (-1), test (-1)
- **XLong** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Fraction of ground-truth items present in the top-10 recommended items.
- `Recall@20` — range: [0, 1]
  - Fraction of ground-truth items present in the top-20 recommended items.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, weighting hits by their position in the ranked list.
- `NDCG@20` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 20, weighting hits by their position in the ranked list.

## Input / output format

**Input**: User interaction history sequence (item IDs) up to a dataset-specific maximum length (50, 200, or 1000).

**Output**: Ranked list of candidate items for the next interaction, evaluated against a single ground-truth item.

## Scoring recipe

```python
import math

def compute_recall_at_k(recommended_list, ground_truth, k):
    hits = sum(1 for item in recommended_list[:k] if item in ground_truth)
    return hits / len(ground_truth)

def compute_ndcg_at_k(recommended_list, ground_truth, k):
    dcg = 0.0
    for i, item in enumerate(recommended_list[:k]):
        if item in ground_truth:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- The paper uses a leave-last-out split rather than random splitting, which is critical for sequential recommendation.
- During testing, validation items must be included in the candidate ranking pool, unlike standard IR benchmarks.
- XLong uses a reduced negative sampling (10k items) for evaluation efficiency, which may slightly bias metrics compared to full ranking.

## Evidence (verbatim from paper)

> For evaluation results, we select models with the best validation Recall@10 scores in training to perform prediction on the test sets. The models are evaluated using Recall@k and NDCG@k metrics, and with $k\in{10,20}$. The predicted items are ranked against all items in the dataset to compute the final scores.

## Citation

```bibtex
@misc{yue2023lru,
  title={Linear Recurrent Units for Sequential Recommendation},
  author={Yue et al. (2023)},
  year={2023},
  note={arXiv:2310.02367}
}
```

- arXiv: 2310.02367