Fgnn Session Rec Eval

---
name: fgnn-session-rec-eval
description: Evaluates session-based recommendation models by predicting the next item in a user's clickstream session. It probes the model's ability to capture latent, non-temporal transition patterns and item order within a session graph. Use when the user wants to benchmark on Yoochoose, Diginetica, or asks about evaluating this task. Reports R@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.11942
  bibtex_key: qiu2019rethinking
  confidence: high
---

# fgnn-session-rec-eval

> Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks — Qiu et al. (2019) (arXiv:1911.11942, 2019)

## What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's clickstream session. It probes the model's ability to capture latent, non-temporal transition patterns and item order within a session graph.

## Datasets

- **Yoochoose** — total ?; splits: 1/64 (-1), 1/4 (-1)
- **Diginetica** — total ?; splits: test (-1)

## Metrics

- `R@20` **(primary)** — range: percent
  - Recall calculated over top-K items. R@K = n_hit / N, where N is the number of test sequences and n_hit counts how many desired items appear in the top K positions of the predicted ranking list.
- `MRR@20` — range: percent
  - Mean Reciprocal Rank calculated over top-K items. MRR@K = (1/N) * sum(1/Rank(v_label)) for v_label in test sequences. The reciprocal is set to 0 if the desired item is not in the top K positions.

## Input / output format

**Input**: A partial session sequence of items [v_s,0, ..., v_s,i-1] of length i (2 ≤ i ≤ n), where the last item v_s,i-1 is the ground-truth target to predict.

**Output**: A ranked list of top-K items (default K=20) predicted as the next item in the session.

## Scoring recipe

```python
def compute_metrics(predictions, gold_items, K=20):
    hits = 0
    reciprocal_ranks = []
    for pred_list, gold in zip(predictions, gold_items):
        if gold in pred_list[:K]:
            hits += 1
            rank = pred_list[:K].index(gold) + 1
            reciprocal_ranks.append(1.0 / rank)
        else:
            reciprocal_ranks.append(0.0)
    recall = (hits / len(gold_items)) * 100
    mrr = (sum(reciprocal_ranks) / len(gold_items)) * 100
    return recall, mrr
```

## Common pitfalls

- Sessions are augmented by splitting each session of length n into n-1 partial sessions of lengths 2 to n, significantly inflating the dataset size compared to standard session-based splits.
- Datasets are pre-filtered to remove sessions of length 1 and items occurring fewer than 5 times, which differs from raw clickstream data and affects baseline comparability if not replicated.
- Metrics are reported as percentages in tables (e.g., 71.12) despite the mathematical formulas using a [0, 1] scale, which can cause confusion when comparing against other papers.

## Evidence (verbatim from paper)

> At a time, a recommender system can give out a few recommended items and a user would choose the first few of them. To keep the same setting as previous baselines, we mainly choose to use top-20 items to evaluate a recommender system and specifically, two metrics, i.e., R@20 and MRR@20. For more detailed comparison, top-5 and top-10 results are considered as well. • R@K (Recall calculated over top-K items). The R@K score is the primary metric that calculates the proportion of test cases which recommend the correct item in a top K position in a ranking list, R@K = n_hit / N, where N represents the number of test sequences S_test in the dataset and n_hit counts the number that the desired items are in the top K position in the ranking list, which is named the hit. • MRR@K (Mean Reciprocal Rank calculated over top-K items). The reciprocal is set to 0 when the desired items are not in the top K position and the calculation is as follows, MRR@K = 1/N sum_{v_label in S_test} 1/Rank(v_label).

## Citation

```bibtex
@misc{qiu2019rethinking,
  title={Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks},
  author={Qiu et al. (2019)},
  year={2019},
  note={arXiv:1911.11942}
}
```

- arXiv: 1911.11942