Session Rec Rnn Eval

---
name: session-rec-rnn-eval
description: Evaluates a model's ability to predict the next item in a user session based on sequential click or watch history. It probes the capability to capture short-term sequential dependencies and maintain context within session-based recommendation scenarios. Use when the user wants to benchmark on RSC15, VIDEO, or asks about evaluating this task. Reports recall@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1511.06939
  bibtex_key: hidasi2015session
  confidence: high
---

# session-rec-rnn-eval

> Session-based Recommendations with Recurrent Neural Networks — Hidasi et al. (2015) (arXiv:1511.06939, 2015)

## What this evaluates

Evaluates a model's ability to predict the next item in a user session based on sequential click or watch history. It probes the capability to capture short-term sequential dependencies and maintain context within session-based recommendation scenarios.

## Datasets

- **RSC15** — total ?; splits: train (7966257), test (15324)
- **VIDEO** — total ?; splits: train (3000000), test (37000)

## Metrics

- `recall@20` **(primary)** — range: [0, 1]
  - Proportion of test cases where the ground-truth next item appears in the model's top-20 ranked recommendations. It does not penalize based on the exact rank within the top-20.
- `MRR@20` — range: [0, 1]
  - Average of reciprocal ranks of the ground-truth items across test cases. The reciprocal rank is set to zero if the item's rank exceeds 20.

## Input / output format

**Input**: A sequence of item IDs representing a user's session history, processed one event at a time.

**Output**: A ranked list of candidate item IDs in descending order of predicted relevance score.

## Scoring recipe

```python
def compute_metrics(predictions, gold_items):
    recall_hits = 0
    mrr_sum = 0.0
    for pred_list, gold in zip(predictions, gold_items):
        if gold in pred_list[:20]:
            recall_hits += 1
        rank = pred_list.index(gold) + 1 if gold in pred_list else 21
        mrr_sum += (1.0 / rank) if rank <= 20 else 0.0
    return recall_hits / len(gold_items), mrr_sum / len(gold_items)
```

## Common pitfalls

- Test set clicks are filtered out if the clicked item was not present in the training set, which can artificially reduce test size for collaborative filtering baselines.
- Sessions of length 1 are removed from both train and test sets, potentially biasing evaluation against models that struggle with single-item contexts.
- For the VIDEO dataset, evaluation is restricted to ranking the target item against only the 30,000 most popular items rather than the full 330,000-item catalog.

## Evidence (verbatim from paper)

> Therefore, our primary evaluation metric is recall@20 that is the proportion of cases having the desired item amongst the top-20 items in all test cases. Recall does not consider the actual rank of the item as long as it is amongst the top-N. ... The second metric used in the experiments is MRR@20 (Mean Reciprocal Rank). That is the average of reciprocal ranks of the desired items. The reciprocal rank is set to zero if the rank is above 20.

## Citation

```bibtex
@misc{hidasi2015session,
  title={Session-based Recommendations with Recurrent Neural Networks},
  author={Hidasi et al. (2015)},
  year={2015},
  note={arXiv:1511.06939}
}
```

- arXiv: 1511.06939