Session Based Rec Eval

---
name: session-based-rec-eval
description: Evaluates a model's ability to predict the next item in a user's shopping session based on sequential item interactions and cross-session collaborative signals. It probes how well the model captures dynamic user interests and leverages historical session data for accurate recommendations. Use when the user wants to benchmark on Diginetica, Tmall, Yoochoose1_64, or asks about evaluating this task. Reports P@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.09593
  bibtex_key: zhang2023cares
  confidence: high
---

# session-based-rec-eval

> Context-aware Session-based Recommendation with Graph Neural Networks — Zhang et al. (2023) (arXiv:2310.09593, 2023)

## What this evaluates

Evaluates a model's ability to predict the next item in a user's shopping session based on sequential item interactions and cross-session collaborative signals. It probes how well the model captures dynamic user interests and leverages historical session data for accurate recommendations.

## Datasets

- **Diginetica** — total 780328; splits: train (719470), test (60858)
- **Tmall** — total 377166; splits: train (351268), test (25898)
- **Yoochoose1_64** — total 425757; splits: train (369859), test (55898)

## Metrics

- `P@20` **(primary)** — range: percent
  - Precision at 20: the fraction of relevant items among the top-20 recommended items.
- `MRR@20` — range: percent
  - Mean Reciprocal Rank at 20: the average reciprocal rank of the first relevant item in the top-20 list.

## Input / output format

**Input**: A sequence of item IDs representing a user's session history.

**Output**: A ranked list of the top-20 candidate items.

## Scoring recipe

```python
def evaluate(predictions, ground_truth, k=20):
    p_at_k = 0
    mrr_at_k = 0
    for pred_list, gt in zip(predictions, ground_truth):
        top_k = pred_list[:k]
        if gt in top_k:
            p_at_k += 1
            rank = top_k.index(gt) + 1
            mrr_at_k += 1.0 / rank
    n = len(predictions)
    return (p_at_k / n) * 100, (mrr_at_k / n) * 100
```

## Common pitfalls

- Strict data filtering (items <5 occurrences, sessions of length 1) and temporal split must be applied exactly as in [8] to match reported numbers.
- Using the full Yoochoose dataset instead of the 1/64 temporal split specified in the paper will yield different results.
- Metrics are reported as percentages (e.g., 56.49) rather than decimals, which can cause confusion when comparing with other works.

## Evidence (verbatim from paper)

> Further, we adopt two widely used evaluation metrics in information retrieval: Precision (P@20) and Mean Reciprocal Rank (MRR@20) for evaluating the performance.

## Citation

```bibtex
@misc{zhang2023cares,
  title={Context-aware Session-based Recommendation with Graph Neural Networks},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2310.09593}
}
```

- arXiv: 2310.09593