Fgnn Sbr Eval

---
name: fgnn-sbr-eval
description: Evaluates session-based recommendation models on e-commerce clickstream data by predicting the next item in a session using graph neural networks and cross-session information. Use when the user wants to benchmark on Yoochoose1/64, Yoochoose1/4, Diginetica, or asks about evaluating this task. Reports R@20, MRR@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.00852
  bibtex_key: qiu2021exploiting
  confidence: high
---

# fgnn-sbr-eval

> Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks — Qiu et al. (2021) (arXiv:2107.00852, 2021)

## What this evaluates

Evaluates session-based recommendation models on e-commerce clickstream data by predicting the next item in a session using graph neural networks and cross-session information.

## Datasets

- **Yoochoose1/64** — total ?; splits: train (369859), test (55898)
- **Yoochoose1/4** — total ?; splits: train (5917746), test (55898)
- **Diginetica** — total ?; splits: train (719470), test (60858)

## Metrics

- `R@20` **(primary)** — range: percent
  - R@K = n_hit / N, where N is the number of test sequences and n_hit counts sequences where the correct item is in the top K.
- `MRR@20` **(primary)** — range: percent
  - MRR@K = (1/N) * Σ(1/Rank(v_label)), assigning 0 if the item is not in top K. Higher scores indicate better ranking quality.

## Input / output format

**Input**: A sequence of item IDs representing a user's click history within a session, truncated to length i.

**Output**: A ranked list of top-K candidate item IDs for the next click in the session.

## Scoring recipe

```python
def evaluate(predictions, golds, K=20):
    hits = 0
    mrr_sum = 0.0
    for pred, gold in zip(predictions, golds):
        if gold in pred[:K]:
            hits += 1
            mrr_sum += 1.0 / (pred.index(gold) + 1)
    N = len(predictions)
    return hits / N, mrr_sum / N
```

## Common pitfalls

- The train/test split uses the most recent 1/64 or 1/4 of the training sequence as the test set, which is a non-standard holdout strategy for session-based recommendation.
- Aggressive filtering removes all sessions of length 1 and items appearing fewer than 5 times, altering the original data distribution and potentially inflating metrics.
- Evaluation assumes a single ground-truth next item per session step, ignoring multi-label recommendations or alternative valid next items in real-world clickstreams.

## Evidence (verbatim from paper)

> For each time, a recommender system can give out a few recommended items and a user would choose the first few of them. To keep the same setting as previous baselines, we mainly choose to use top-20 items to evaluate a recommender system and specifically, two metrics, i.e., R@20 and MRR@20. For more detailed comparison, top-5 and top-10 results are considered as well.

## Citation

```bibtex
@misc{qiu2021exploiting,
  title={Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks},
  author={Qiu et al. (2021)},
  year={2021},
  note={arXiv:2107.00852}
}
```

- arXiv: 2107.00852