Narm Session Rec Eval

---
name: narm-session-rec-eval
description: Evaluates session-based recommendation models by predicting the next item a user will click based on their sequential interaction history within a session. It probes the model's ability to capture both sequential behavior and session-level intent/purpose. Use when the user wants to benchmark on YOOCHOOSE 1/64, YOOCHOOSE 1/4, DIGINETICA, or asks about evaluating this task. Reports Recall@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1711.04725
  bibtex_key: li2017narm
  confidence: high
---

# narm-session-rec-eval

> Neural Attentive Session-based Recommendation — Jing Li et al. (2017) (arXiv:1711.04725, 2017)

## What this evaluates

Evaluates session-based recommendation models by predicting the next item a user will click based on their sequential interaction history within a session. It probes the model's ability to capture both sequential behavior and session-level intent/purpose.

## Datasets

- **YOOCHOOSE 1/64** — total ?; splits: train (369859), test (55898)
- **YOOCHOOSE 1/4** — total ?; splits: train (5917746), test (55898)
- **DIGINETICA** — total ?; splits: train (719470), test (60858)

## Metrics

- `Recall@20` **(primary)** — range: percent
  - The proportion of test cases where the ground-truth next item appears in the top-20 recommended items. It does not consider the actual rank as long as the item is within the top-N.
- `MRR@20` — range: percent
  - The average of reciprocal ranks of the desired items across all test cases. The reciprocal rank is set to zero if the item's rank exceeds 20.

## Input / output format

**Input**: A sequence of item IDs representing a user's click history within a session, formatted as [x_1, x_2, ..., x_{n-1}].

**Output**: A ranked list of the top-20 candidate items for the next click.

## Scoring recipe

```python
recalls = []
mrrs = []
for pred_list, gold in test_set:
    top20 = pred_list[:20]
    recalls.append(1.0 if gold in top20 else 0.0)
    if gold in top20:
        rank = top20.index(gold) + 1
        mrrs.append(1.0 / rank)
    else:
        mrrs.append(0.0)
recall_20 = sum(recalls) / len(test_set) * 100
mrr_20 = sum(mrrs) / len(test_set) * 100
```

## Common pitfalls

- YOOCHOOSE is split by time into 1/64 and 1/4 training fractions, which reduces item vocabulary overlap with the test set compared to using the full training set.
- DIGINETICA evaluation filters out test clicks where the target item is absent from the training set, while YOOCHOOSE evaluation retains them to match baseline settings, creating inconsistent strictness across datasets.
- MRR@20 truncates the reciprocal rank to 0 for items ranked beyond position 20, potentially underestimating performance for items ranked just above the cutoff.

## Evidence (verbatim from paper)

> Recall@20: The primary evaluation metric is Recall@20 that is the proportion of cases when the desired item is amongst the top-20 items in all test cases. Recall@N does not consider the actual rank of the item as long as it is amongst the top-N and also usually correlates well with other metrics such as click-through rate (CTR) [21]. MRR@20 (Mean Reciprocal Rank), which is the average of reciprocal ranks of the desire items. The reciprocal rank is set to zero if the rank is larger than 20.

## Citation

```bibtex
@misc{li2017narm,
  title={Neural Attentive Session-based Recommendation},
  author={Jing Li et al. (2017)},
  year={2017},
  note={arXiv:1711.04725}
}
```

- arXiv: 1711.04725