Context Tree Session Rec Eval

---
name: context-tree-session-rec-eval
description: This benchmark evaluates session-based recommendation models by predicting the immediate next item in a user's interaction sequence. It probes the model's ability to capture sequential dependencies and adapt to evolving user preferences and new items in both static and continuously updating environments. Use when the user wants to benchmark on MOOC, News, and RecSys Challenge Datasets, or asks about evaluating this task. Reports HR@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.03733
  bibtex_key: mi2018contexttree
  confidence: high
---

# context-tree-session-rec-eval

> Context Tree for Adaptive Session-based Recommendation — Mi et al. (2018) (arXiv:1806.03733, 2018)

## What this evaluates

This benchmark evaluates session-based recommendation models by predicting the immediate next item in a user's interaction sequence. It probes the model's ability to capture sequential dependencies and adapt to evolving user preferences and new items in both static and continuously updating environments.

## Datasets

- **MOOC, News, and RecSys Challenge Datasets** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@k` **(primary)** — range: [0, 1]
  - Averaged hit rate (recall) indicating whether the ground-truth next item appears in the top-k recommended items. Calculated as the mean of 1 if gold in top-k else 0 across all test instances.
- `MRR@k` — range: [0, 1]
  - Averaged mean reciprocal rank of the ground-truth next item within the top-k recommended list. Calculated as the mean of 1/rank if gold in top-k else 0 across all test instances.

## Input / output format

**Input**: A sequence of previously interacted items in a session (e.g., [item_1, item_2, ..., item_t]).

**Output**: A ranked list of k candidate items predicted as the next item to interact with.

## Scoring recipe

```python
def compute_metrics(predictions, golds, k=20):
    hr_scores, mrr_scores = [], []
    for pred_list, gold in zip(predictions, golds):
        top_k = pred_list[:k]
        if gold in top_k:
            hr_scores.append(1.0)
            rank = top_k.index(gold) + 1
            mrr_scores.append(1.0 / rank)
        else:
            hr_scores.append(0.0)
            mrr_scores.append(0.0)
    return {'HR@k': sum(hr_scores) / len(hr_scores), 'MRR@k': sum(mrr_scores) / len(mrr_scores)}
```

## Common pitfalls

- Static evaluation uses a fixed train/test split where the test set only contains items seen during training, whereas adaptive evaluation updates the model continuously as new events arrive, introducing unseen items.
- Preprocessing rules differ by dataset domain: MOOC and News datasets retain duplicate items within sessions, while RecSys Challenge datasets remove duplicates and filter out items with fewer than 5 total views.
- In the adaptive setting, GRU4Rec is updated only after a full session completes rather than per-event, which may unfairly penalize it compared to nonparametric methods that update incrementally.

## Evidence (verbatim from paper)

> The recommendation model generates a ranked list of k items as prediction for each testing interaction event, and the evaluation metrics that we are going to use across later experiments are as follow: - HR@k: The averaged hit rates (recall) of having the desired items amongst the top-k recommended item lists. - MRR@k: The averaged mean reciprocal ranks of the desired items in top-k recommended item lists.

## Citation

```bibtex
@misc{mi2018contexttree,
  title={Context Tree for Adaptive Session-based Recommendation},
  author={Mi et al. (2018)},
  year={2018},
  note={arXiv:1806.03733}
}
```

- arXiv: 1806.03733