Diffurec Eval

---
name: diffurec-eval
description: Evaluates a model's ability to predict the next item in a user's interaction sequence by capturing dynamic preferences and multi-aspect item representations. It probes sequential recommendation performance under varying sequence lengths and item popularities. Use when the user wants to benchmark on Amazon Beauty, Amazon Toys, Movielens-1M, Steam, or asks about evaluating this task. Reports HR@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.00686
  bibtex_key: li2023diffurec
  confidence: high
---

# diffurec-eval

> DiffuRec: A Diffusion Model for Sequential Recommendation — Li et al. (2023) (arXiv:2304.00686, 2023)

## What this evaluates

Evaluates a model's ability to predict the next item in a user's interaction sequence by capturing dynamic preferences and multi-aspect item representations. It probes sequential recommendation performance under varying sequence lengths and item popularities.

## Datasets

- **Amazon Beauty** — total 22363; splits: train (-1), val (-1), test (-1)
- **Amazon Toys** — total 19412; splits: train (-1), val (-1), test (-1)
- **Movielens-1M** — total 6040; splits: train (-1), val (-1), test (-1)
- **Steam** — total 281428; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@K` **(primary)** — range: [0, 1]
  - Hit Rate at K. Returns 1 if the target item appears in the top-K ranked list, else 0. Averaged across all test instances.
- `NDCG@K` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K. Returns 1/log2(rank+1) if the target item is in the top-K list, else 0. Averaged across all test instances.

## Input / output format

**Input**: Chronologically ordered sequence of user-item interactions (items i_1 to i_{n-2}) used to predict the next item i_n.

**Output**: A ranked list of all candidate items, from which the top-K predictions are extracted.

## Scoring recipe

```python
def compute_metrics(ranked_items, target_item, K):
    hit = 1.0 if target_item in ranked_items[:K] else 0.0
    ndcg = 0.0
    if hit:
        rank = ranked_items.index(target_item) + 1
        if rank <= K:
            ndcg = 1.0 / math.log2(rank + 1)
    return hit, ndcg
```

## Common pitfalls

- Using negative sampling instead of ranking all candidate items, which the paper explicitly notes causes inconsistency when the number of negative items is small.
- Ignoring the chronological leave-one-out split protocol, which is critical for temporal sequential recommendation evaluation.
- Not applying the dataset-specific maximum sequence length limits (50 for Beauty/Toys/Steam, 200 for MovieLens-1M) during preprocessing.

## Evidence (verbatim from paper)

> Following the common data preprocessing method [25, 42, 48], we treat all reviews or ratings as implicit feedback (i.e., a user-item interaction) and chronologically organize them by their timestamps. ... we adopt a leave-one-out strategy for performance evaluation. To be specific, for all the datasets, given a sequence S = {i1, i2, ..., in}, we utilize the most recent interaction (in) for testing, the penultimate interaction (in-1) for model validation, and the earlier ones {(i1, i2, ..., in-2)} for model training. ... As for the evaluation metric, we evaluate all models with HR@K (Hit Rate) and NDCG@K (Normalized Discounted Cumulative Gain). We report the experimental results with K = {5, 10, 20}. HR@K represents the proportion of the hits recommended among the top-K list. NDCG@K further evaluates the ranking performance by considering the ranking positions of these hits. The NDCG@K is set to 0 when the rank exceeds K. Because the evaluation with sampling may cause inconsistency when the number of negative items is small [26], we rank all candidate items for target item prediction.

## Citation

```bibtex
@misc{li2023diffurec,
  title={DiffuRec: A Diffusion Model for Sequential Recommendation},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2304.00686}
}
```

- arXiv: 2304.00686