Globo Session Rec Eval

---
name: globo-session-rec-eval
description: This benchmark evaluates session-based news recommendation systems by predicting the next article a user will click based on their recent interaction history. It probes a model's ability to capture short-term user intent, handle temporal dynamics, and leverage both content and contextual features in a streaming environment. Use when the user wants to benchmark on Globo.com, or asks about evaluating this task. Reports HR@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.00076
  bibtex_key: moreira2018news
  confidence: high
---

# globo-session-rec-eval

> News Session-Based Recommendations using Deep Neural Networks — Moreira et al. (2018) (arXiv:1808.00076, 2018)

## What this evaluates

This benchmark evaluates session-based news recommendation systems by predicting the next article a user will click based on their recent interaction history. It probes a model's ability to capture short-term user intent, handle temporal dynamics, and leverage both content and contextual features in a streaming environment.

## Datasets

- **Globo.com** — total ?; splits: train (-1), test (-1); repo https://github.com/gabrielspmoreira/chameleon_recsys

## Metrics

- `HR@5` **(primary)** — range: [0, 1]
  - 1 if the ground-truth next item is in the top-5 predicted items, else 0. Averaged over all sessions.
- `MRR@5` — range: [0, 1]
  - 1/r, where r is the rank of the ground-truth next item in the top-5 list (or 0 if not in top-5). Averaged over all sessions.

## Input / output format

**Input**: A session represented as a sequence of clicked article IDs (padded to batch max length), augmented with per-article features (content embedding, recent popularity, recency) and user context (platform, device type).

**Output**: A ranked list of candidate articles (ground-truth next item plus sampled negatives) for next-click prediction.

## Scoring recipe

```python
def score(predictions, gold):
    hr5 = 1.0 if gold in predictions[:5] else 0.0
    mrr5 = 0.0
    if gold in predictions[:5]:
        rank = predictions.index(gold) + 1
        mrr5 = 1.0 / rank
    return {'HR@5': hr5, 'MRR@5': mrr5}
```

## Common pitfalls

- Temporal leakage: Random train/test splits ignore the time-sensitive nature of news; this benchmark strictly enforces hour-by-hour temporal splits to prevent future data from leaking into training.
- Fresh item handling: Models like GRU4Rec cannot recommend unseen items, so evaluations must explicitly ignore clicks on articles published after the last training window (~2% of clicks), which can artificially inflate baseline scores.
- Negative sampling bias: The evaluation uses a specific negative sampling strategy (items from the same batch or recent global buffer); changing this strategy significantly alters the difficulty and reported metrics.

## Evidence (verbatim from paper)

> The Top-N evaluation metrics used in this study were Hit Rate (HR@5) *(Ludewig and Jannach, 2018)* which checks whether the clicked item is present in the top-5 ranked items, and Mean Reciprocal Rank (MRR@5) *(Hidasi et al., 2016a)* *(Jugovac et al., 2018)* *(Ludewig and Jannach, 2018)*, a ranking metric, sensitive to the position of clicked item, which assigns higher score at top ranks.

## Citation

```bibtex
@misc{moreira2018news,
  title={News Session-Based Recommendations using Deep Neural Networks},
  author={Moreira et al. (2018)},
  year={2018},
  note={arXiv:1808.00076}
}
```

- arXiv: 1808.00076