Long Tail Session Rec Eval

---
name: long-tail-session-rec-eval
description: This evaluation probes a session-based recommendation model's ability to accurately predict the next item in a user's interaction sequence while mitigating popularity bias. It measures both standard ranking accuracy and the model's capacity to recommend long-tail items, ensuring recommendations align with user-specific item distribution preferences rather than just global popularity. Use when the user wants to benchmark on YOOCHOOSE, Last.fm, or asks about evaluating this task. Reports Recall@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.02581
  bibtex_key: chen2021longtail
  confidence: high
---

# long-tail-session-rec-eval

> Long-Tail Session-based Recommendation from Calibration — Chen et al. (2021) (arXiv:2112.02581, 2021)

## What this evaluates

This evaluation probes a session-based recommendation model's ability to accurately predict the next item in a user's interaction sequence while mitigating popularity bias. It measures both standard ranking accuracy and the model's capacity to recommend long-tail items, ensuring recommendations align with user-specific item distribution preferences rather than just global popularity.

## Datasets

- **YOOCHOOSE** — total ?; splits: train (-1), val (-1), test (-1)
- **Last.fm** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Recall@20` **(primary)** — range: [0, 1]
  - Computes the proportion of sessions where the ground-truth next item appears in the top-20 recommended items. Formula: (1/|S_te|) * sum(1(x_{n+1} in RL_s)).
- `MRR@20` — range: [0, 1]
  - Measures the mean reciprocal rank of the ground-truth item within the top-20 list. If the item is not in the top-20, the score is 0. Formula: (1/|S_te|) * sum(1/rank(x_{n+1}, RL_s)).
- `Coverage@20` — range: [0, 1]
  - Represents the proportion of unique items recommended across all test sessions relative to the entire item catalog. Formula: |union(RL_s)| / |I|.
- `TailCoverage@20` — range: [0, 1]
  - Similar to Coverage but restricted to long-tail items. Measures the proportion of tail items in the catalog that appear in the top-20 recommendations. Formula: |union(RL_s intersect I_Tail)| / |I_Tail|.
- `Tail@20` — range: [0, 1]
  - Calculates the average proportion of tail items in each session's top-20 recommendation list. Formula: (1/|S_te|) * sum(|RL_s intersect I_Tail| / N).
- `IP` — range: percent
  - Improvement Percentage relative to a baseline model. Formula: (Metric_ourmethod - Metric_othermethod) / Metric_othermethod.

## Input / output format

**Input**: A session sequence of user-item interactions (historical items) used to predict the next item.

**Output**: A ranked list of top-N (N=20) recommended items.

## Scoring recipe

```python
def compute_metrics(predictions, gold_items, tail_items, all_items, N=20):
    recalls, mrrs, cov_items, tail_cov_items, tail_ratios = [], [], set(), set(), []
    for pred_list, gold in zip(predictions, gold_items):
        top_n = pred_list[:N]
        recalls.append(1.0 if gold in top_n else 0.0)
        rank = next((i+1 for i, item in enumerate(top_n) if item == gold), N+1)
        mrrs.append(1.0/rank if rank <= N else 0.0)
        cov_items.update(top_n)
        tail_cov_items.update(top_n & tail_items)
        tail_ratios.append(len(top_n & tail_items) / N)
    return {
        'Recall@20': sum(recalls) / len(recalls),
        'MRR@20': sum(mrrs) / len(mrrs),
        'Coverage@20': len(cov_items) / len(all_items),
        'TailCoverage@20': len(tail_cov_items) / len(tail_items),
        'Tail@20': sum(tail_ratios) / len(tail_ratios)
    }
```

## Common pitfalls

- Tail items are typically defined by purchase frequency (e.g., items with <2 purchases), but the exact threshold is not specified in the text, leading to inconsistent tail definitions across studies.
- The IP (Improvement Percentage) metric is relative to the chosen baseline, so reporting it without explicitly stating the baseline model makes cross-paper comparisons ambiguous.
- Using N=20 for all metrics is fixed here, but many recommendation papers use N=10 or N=50, causing direct numerical comparison issues if not normalized or explicitly noted.

## Evidence (verbatim from paper)

> Following previous work $ ^{[10, 13]} $ , we use Recall and MRR as evaluation metrics. • Recall@N (Rec@N) is a widely used metric in recommendation and information retrieval areas. Recall@N computes the proportion of correct items in the top-N items of the list. $$ Recall@N=\frac{1}{\|S_{te}\|}\sum_{s\in S_{te}}1(x_{n+1}\in RL_{s}) $$

## Citation

```bibtex
@misc{chen2021longtail,
  title={Long-Tail Session-based Recommendation from Calibration},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2112.02581}
}
```

- arXiv: 2112.02581