Dgser Eval

---
name: dgser-eval
description: Evaluates a model's ability to perform sequential next-item recommendation by modeling dynamic collaborative signals and temporal user preferences. It tests how well the system captures high-order item transitions and time-annotated graph structures to predict the next interaction in a user's history. Use when the user wants to benchmark on Amazon-CDs, Amazon-Games, Amazon-Beauty, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.07368
  bibtex_key: zhang2021dgser
  confidence: high
---

# dgser-eval

> Dynamic Graph Neural Networks for Sequential Recommendation — Zhang et al. (2021) (arXiv:2104.07368, 2021)

## What this evaluates

Evaluates a model's ability to perform sequential next-item recommendation by modeling dynamic collaborative signals and temporal user preferences. It tests how well the system captures high-order item transitions and time-annotated graph structures to predict the next interaction in a user's history.

## Datasets

- **Amazon-CDs** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Games** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Beauty** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list by discounting the relevance of the ground-truth item based on its position, normalized by the ideal DCG.
- `Hit@10` — range: [0, 1]
  - Binary metric indicating whether the ground-truth item appears in the top-10 recommended items.

## Input / output format

**Input**: A user's historical interaction sequence with associated timestamps, used to predict the next item in the sequence.

**Output**: A ranked list of top-10 candidate items (including the ground-truth item and 100 randomly sampled negatives).

## Scoring recipe

```python
# For each test user:
# 1. Sample 100 negative items uniformly at random.
# 2. Combine with ground-truth item to form a candidate set of 101 items.
# 3. Score each candidate using the model and sort descending.
# 4. Find rank of ground-truth item (1-indexed).
# Hit@10 = 1 if rank <= 10 else 0
# DCG@10 = 1 / log2(rank + 1)
# NDCG@10 = DCG@10 / 1 (ideal DCG for single relevant item at rank 1 is 1)
# Average Hit@10 and NDCG@10 over all test samples.
```

## Common pitfalls

- Negative sampling is fixed at 100 items per test sample, which differs from the full-item ranking used in some other recommendation benchmarks.
- Datasets are filtered to only include users and items with at least 5 interactions, which significantly reduces sparsity compared to raw Amazon dumps.
- Sequences are split chronologically (last item=test, second last=val, rest=train), not randomly shuffled.

## Evidence (verbatim from paper)

> We adopt two widely-used metrics Hit@$K$ and NDCG@$K$, to evaluate all methods. Hit@$K$ indicates the proportion of the ground-truth items among the top@$K$ items, while NDCG@$K$ is position-aware metric, and higher NDCG means target items tend to have more top rank positions. Following *[[23](#bib.bib23 ""), [11](#bib.bib11 "")]*, for each test sample, we randomly sample 100 negative items, and rank these items with the ground-truth item. We evaluate Hit@$K$ and NDCG@$K$ based on these 101 items. By default, we set $K$\=10.

## Citation

```bibtex
@misc{zhang2021dgser,
  title={Dynamic Graph Neural Networks for Sequential Recommendation},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2104.07368}
}
```

- arXiv: 2104.07368