Sbr Srgi Eval

---
name: sbr-srgi-eval
description: Evaluates session-based recommendation models by predicting the next item in a user's session sequence. It probes the model's ability to capture sequential item transitions and leverage global item-transition patterns across sessions to improve ranking accuracy. Use when the user wants to benchmark on Diginetica, Tmall, Nowplaying, or asks about evaluating this task. Reports P@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.10173
  bibtex_key: wang2020exploring
  confidence: high
---

# sbr-srgi-eval

> Exploring Global Information for Session-based Recommendation — Ziyang Wang et al. (arXiv:2011.10173, 2020)

## What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's session sequence. It probes the model's ability to capture sequential item transitions and leverage global item-transition patterns across sessions to improve ranking accuracy.

## Datasets

- **Diginetica** — total 780328; splits: train (719470), test (60858)
- **Tmall** — total 377166; splits: train (351268), test (25898)
- **Nowplaying** — total 915128; splits: train (825304), test (89824)

## Metrics

- `P@20` **(primary)** — range: [0, 1]
  - P@N = n_hit / n_test, where n_hit is the number of target items in the top N recommended list and n_test is the total number of test instances.
- `MRR@20` — range: [0, 1]
  - MRR@N = (1/n_test) * Σ(1/Rank(v_target)), where the reciprocal rank is set to zero if the rank exceeds N.

## Input / output format

**Input**: A session sequence of items [s1, s2, ..., sn] representing a user's interaction history.

**Output**: A ranked list of top-20 candidate items predicted as the next item in the session.

## Scoring recipe

```python
def compute_metrics(predictions, test_items):
    p_hits = 0
    mrr_sum = 0.0
    n_test = len(test_items)
    for pred, true_item in zip(predictions, test_items):
        if true_item in pred[:20]:
            p_hits += 1
            rank = pred.index(true_item) + 1
            mrr_sum += 1.0 / rank
    p_at_20 = p_hits / n_test
    mrr_at_20 = mrr_sum / n_test
    return p_at_20, mrr_at_20
```

## Common pitfalls

- Hyperparameters are tuned on a random 10% subset of the training set rather than a dedicated validation split.
- Sequence splitting creates overlapping training and test instances, which can inflate performance if not strictly separated.
- Datasets undergo strict filtering (sessions length 1 removed, items <5 occurrences removed, Tmall sessions >40 removed), differing from raw data distributions.

## Evidence (verbatim from paper)

> We adopt two widely used ranking based metrics: P@N and MRR@N by following previous work. P@N(Precision): The P@N score is typically used as a measure of accuracy. It represents the proportion of correctly recommended items in top N recommended item list, which is defined as: P@N = n_hit / n_test, where n_test denotes the number of test data and n_hit denotes the number of the target items appearing in of top N recommended items. MRR@N(Mean Reciprocal Rank): The MRR@N score is the average of reciprocal rank of the correctly-recommended items. The reciprocal rank is set to zero if the rank exceeds N, MRR@N = (1/n_test) Σ(1/Rank(v_target)). Here, we choose N=20 for both P@N and MRR@N, as recommendation systems should focus on top ranked items.

## Citation

```bibtex
@misc{wang2020exploring,
  title={Exploring Global Information for Session-based Recommendation},
  author={Ziyang Wang et al.},
  year={2020},
  note={arXiv:2011.10173}
}
```

- arXiv: 2011.10173