Samrec Eval

---
name: samrec-eval
description: Evaluates sequential recommendation models on their ability to predict the next item in a user's chronological interaction history. It specifically probes whether sharpness-aware minimization improves generalization and data efficiency compared to standard Transformers and self-supervised baselines. Use when the user wants to benchmark on Amazon-Beauty, Amazon-Sports, Amazon-Toys, Yelp, or asks about evaluating this task. Reports HR@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.10347
  bibtex_key: lai2023enhancing
  confidence: high
---

# samrec-eval

> Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation — Lai et al. (2023) (arXiv:2308.10347, 2023)

## What this evaluates

Evaluates sequential recommendation models on their ability to predict the next item in a user's chronological interaction history. It specifically probes whether sharpness-aware minimization improves generalization and data efficiency compared to standard Transformers and self-supervised baselines.

## Datasets

- **Amazon-Beauty** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Sports** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Toys** — total ?; splits: train (-1), val (-1), test (-1)
- **Yelp** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@10` **(primary)** — range: [0, 1]
  - 1 if the ground-truth item appears in the top-10 predicted items, else 0.
- `NDCG@10` — range: [0, 1]
  - Sum of 1/log2(rank+1) for each hit in the top-10 list, normalized by the ideal DCG@10 (which equals 1 for a single positive item).

## Input / output format

**Input**: A chronological sequence of user-item interaction IDs.

**Output**: A ranked list of top-10 candidate item IDs.

## Scoring recipe

```python
def evaluate(predictions, ground_truth):
    hr = 1.0 if ground_truth in predictions else 0.0
    if ground_truth in predictions:
        rank = predictions.index(ground_truth) + 1
        ndcg = 1.0 / math.log2(rank + 1)
    else:
        ndcg = 0.0
    return hr, ndcg
```

## Common pitfalls

- Failing to apply the 5-core filtering before splitting the data.
- Not sorting user interactions by timestamp before creating train/val/test splits.
- Reporting results from a single run instead of averaging over 10 independent runs.

## Evidence (verbatim from paper)

> We use two widely used metrics, Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG), to evaluate performance. We repeat the experiments 10 times independently and report the average results for HR@10 and NDCG@10.

## Citation

```bibtex
@misc{lai2023enhancing,
  title={Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation},
  author={Lai et al. (2023)},
  year={2023},
  note={arXiv:2308.10347}
}
```

- arXiv: 2308.10347