Sequential Recsys Eval

---
name: sequential-recsys-eval
description: Evaluates the performance and reproducibility of sequential recommender system (SRS) models across multiple user-item interaction datasets. It probes how architectural choices, hyperparameter settings, and training configurations affect ranking metrics and computational emissions. Use when the user wants to benchmark on Beauty, FS-NYC, FS-TKY, ML-100k, ML-1M, ML-20M, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.03873
  bibtex_key: betello2024reproducible
  confidence: high
---

# sequential-recsys-eval

> A Reproducible Analysis of Sequential Recommender Systems — Betello et al. (2024) (arXiv:2408.03873, 2024)

## What this evaluates

Evaluates the performance and reproducibility of sequential recommender system (SRS) models across multiple user-item interaction datasets. It probes how architectural choices, hyperparameter settings, and training configurations affect ranking metrics and computational emissions.

## Datasets

- **Beauty** — total 7113; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
- **FS-NYC** — total 179468; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
- **FS-TKY** — total 494807; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
- **ML-100k** — total 99287; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
- **ML-1M** — total 999611; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
- **ML-20M** — total 19984024; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Calculated as DCG@10 divided by the ideal DCG@10, where DCG discounts relevance by the logarithm of the rank position.
- `Recall@20` — range: [0, 1]
  - Proportion of relevant items found in the top-20 recommended items out of all relevant items in the ground truth.
- `Precision` — range: [0, 1]
  - Proportion of recommended items in the top-K list that are relevant.
- `MAP` — range: [0, 1]
  - Mean Average Precision across all users, averaging the precision at each relevant item's rank position.

## Input / output format

**Input**: User-item interaction sequences (lists of item IDs) representing a user's historical behavior, processed to generate predictions for the next item(s).

**Output**: Ranked list of candidate items (or top-K recommendations) for the next interaction, evaluated against ground-truth next items.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k=10):
    # predictions: list of item IDs ranked by model
    # ground_truth: list of relevant item IDs (usually 1 for next-item rec)
    # Recall@k
    recall = len(set(predictions[:k]) & set(ground_truth)) / max(len(ground_truth), 1)
    # NDCG@k
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(predictions[:k]) if item in ground_truth)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return ndcg, recall
```

## Common pitfalls

- Using different loss functions (e.g., BPR vs BCE) or training durations across models, which heavily skews performance rankings.
- Inconsistent data splitting (e.g., random vs temporal last-interaction split) or negative sampling strategies.
- Ignoring computational emissions and training time, which are critical for fair comparison in modern SRS evaluations.

## Evidence (verbatim from paper)

> To evaluate the performance of sequential recommendation algorithms, we use four widely used metric, also common in Information Retrieval (IR): Precision, Recall, NDCG and MAP. ... For testing, as in (Sun et al., [2019]; Kang and McAuley, [2018]), we keep the last interaction for each user, while for the validation set, the second to last action is retained. All remaining interactions contribute to the training set.

## Citation

```bibtex
@misc{betello2024reproducible,
  title={A Reproducible Analysis of Sequential Recommender Systems},
  author={Betello et al. (2024)},
  year={2024},
  note={arXiv:2408.03873}
}
```

- arXiv: 2408.03873