Session Aware Rec Eval

---
name: session-aware-rec-eval
description: Evaluates the predictive performance of session-based and session-aware recommendation models in ranking the next item a user will interact with. It benchmarks both neural and non-neural approaches across multiple real-world interaction datasets to assess accuracy, coverage, and popularity bias. Use when the user wants to benchmark on RETAIL, XING, COSMETICS, LASTFM, or asks about evaluating this task. Reports MAP@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.03424
  bibtex_key: [REDACTED]
  confidence: high
---

# session-aware-rec-eval

> Session-aware Recommendation: A Surprising Quest for the State-of-the-art — Latifi et al. (2020) (arXiv:2011.03424, 2020)

## What this evaluates

Evaluates the predictive performance of session-based and session-aware recommendation models in ranking the next item a user will interact with. It benchmarks both neural and non-neural approaches across multiple real-world interaction datasets to assess accuracy, coverage, and popularity bias.

## Datasets

- **RETAIL** — total ?; splits: test (-1)
- **XING** — total ?; splits: test (-1)
- **COSMETICS** — total ?; splits: test (-1)
- **LASTFM** — total ?; splits: test (-1)

## Metrics

- `MAP@20` **(primary)** — range: [0, 1]
  - Mean Average Precision at cutoff 20; averages precision scores at ranks where relevant items appear in the top-20 recommendations.
- `P@20` — range: [0, 1]
  - Precision at 20; fraction of top-20 recommended items that are relevant.
- `R@20` — range: [0, 1]
  - Recall at 20; fraction of all relevant items in the session that appear in the top-20 recommendations.
- `HR@20` — range: [0, 1]
  - Hit Rate at 20; binary indicator (1 if at least one relevant item is in top-20, else 0).
- `MRR@20` — range: [0, 1]
  - Mean Reciprocal Rank at 20; inverse of the rank of the first relevant item in the top-20 list.
- `COV@20` — range: [0, 1]
  - Coverage at 20; fraction of distinct items recommended across the top-20 lists.
- `POP@20` — range: other
  - Popularity at 20; average popularity (interaction frequency) of recommended items in the top-20 lists.

## Input / output format

**Input**: A sequence of items interacted by a user in a session, optionally augmented with long-term user profile data for session-aware models.

**Output**: A ranked list of up to 20 recommended items.

## Scoring recipe

```python
def evaluate(preds, gold, k=20):
    hits = sum(1 for item in preds[:k] if item in gold)
    precision = hits / k
    recall = hits / len(gold) if gold else 0.0
    hr = 1.0 if hits > 0 else 0.0
    mrr = sum(1.0 / (i + 1) for i, item in enumerate(preds[:k]) if item in gold)
    ap = sum(precision for i, item in enumerate(preds[:k]) if item in gold) / min(len(gold), k)
    return {'P@20': precision, 'R@20': recall, 'HR@20': hr, 'MRR@20': mrr, 'MAP@20': ap}
```

## Common pitfalls

- Using weak or improperly optimized baselines that fail to match the original authors' reported performance.
- Comparing session-based and session-aware models without properly accounting for long-term preference information or using inconsistent evaluation protocols.
- Neglecting statistical significance testing (e.g., Kruskal-Wallis and Wilcoxon signed-rank tests) when reporting performance differences.

## Evidence (verbatim from paper)

> Tables 3–6 show the results of our performance comparison of neural and non-neural methods, ordered by the values obtained for the MAP@20 metric. Here, we correspondingly report the values obtained by applying a cut-off threshold of 20.

## Citation

```bibtex
@misc{latifi2020session,
  title={Session-aware Recommendation: A Surprising Quest for the State-of-the-art},
  author={Latifi et al. (2020)},
  year={2020},
  note={arXiv:2011.03424}
}
```

- arXiv: 2011.03424