Sequential Rec Diffrec Eval

---
name: sequential-rec-diffrec-eval
description: Evaluates a model's ability to predict the next item in a user's sequential interaction history. It probes how well the system captures temporal user preferences and handles discrete recommendation data under a strict chronological split. Use when the user wants to benchmark on Amazon Beauty, Amazon Toys, MovieLens-1M, or asks about evaluating this task. Reports NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.04541
  bibtex_key: du2023sequentialrecdiffrec
  confidence: high
---

# sequential-rec-diffrec-eval

> Sequential Recommendation with Diffusion Models — Du et al. (2023) (arXiv:2304.04541, 2023)

## What this evaluates

Evaluates a model's ability to predict the next item in a user's sequential interaction history. It probes how well the system captures temporal user preferences and handles discrete recommendation data under a strict chronological split.

## Datasets

- **Amazon Beauty** — total ?; splits: train/val/test (leave-one-out) (-1)
- **Amazon Toys** — total ?; splits: train/val/test (leave-one-out) (-1)
- **MovieLens-1M** — total ?; splits: train/val/test (leave-one-out) (-1)

## Metrics

- `HR@K` — range: [0, 1]
  - Hit Ratio at K: the fraction of users for whom the true next item appears in the top-K ranked recommendations. K ∈ {5, 10, 20}.
- `NDCG@K` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K: measures ranking quality by discounting the gain logarithmically with rank position. Since only one item is relevant per user, NDCG@K = 1/log2(rank+1) if the item is in top-K, else 0. K ∈ {5, 10, 20}.

## Input / output format

**Input**: Chronologically sorted sequence of user-item interactions (implicit feedback) up to the penultimate item.

**Output**: A ranked list of top-K candidate items from the entire item catalog.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k):
    top_k = predictions[:k]
    hr = 1.0 if gold in top_k else 0.0
    rank = top_k.index(gold) + 1 if gold in top_k else k + 1
    dcg = 1.0 / math.log2(rank + 1)
    idcg = 1.0 / math.log2(2)
    ndcg = dcg / idcg
    return hr, ndcg
```

## Common pitfalls

- Ranking on a sampled negative set instead of the full item catalog leads to unfair comparisons and inflated scores.
- Using random train/val/test splits instead of the chronological leave-one-out strategy breaks the sequential recommendation protocol.
- Treating explicit ratings as implicit feedback without filtering low-activity users/items (<5 interactions) alters dataset statistics.

## Evidence (verbatim from paper)

> To evaluate the performance of sequential recommenders, we choose the score of top-K Hit Ratio (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K) as the evaluation metrics, where K ∈ {5, 10, 20}. The leave-one-out evaluation strategy is adopted, leaving out the last item for test, the second-to-last item for validation, and the rest for training. As sampled metrics might lead to unfair comparisons [28], we rank the prediction results on the whole dataset without negative sampling.

## Citation

```bibtex
@misc{du2023sequentialrecdiffrec,
  title={Sequential Recommendation with Diffusion Models},
  author={Du et al. (2023)},
  year={2023},
  note={arXiv:2304.04541}
}
```

- arXiv: 2304.04541