Pen4rec Eval

---
name: pen4rec-eval
description: Evaluates a model's ability to predict the next item in a session-based recommendation task by capturing evolving user preferences and mitigating preference drift over time. Use when the user wants to benchmark on Yoochoose, Diginetica, LastFM, PHEME, or asks about evaluating this task. Reports P@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.09306
  bibtex_key: dou2021pen4rec
  confidence: high
---

# pen4rec-eval

> PEN4Rec: Preference Evolution Networks for Session-based Recommendation — Dou Hu et al. (arXiv:2106.09306, 2021)

## What this evaluates

Evaluates a model's ability to predict the next item in a session-based recommendation task by capturing evolving user preferences and mitigating preference drift over time.

## Datasets

- **Yoochoose** — total ?; splits: train (-1), val (-1), test (-1)
- **Diginetica** — total ?; splits: train (-1), val (-1), test (-1)
- **LastFM** — total ?; splits: train (-1), val (-1), test (-1)
- **PHEME** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `P@20` **(primary)** — range: percent
  - Precision at rank 20. Calculated as the fraction of test sessions where the ground-truth next item appears in the top 20 recommended items.
- `MRR@20` — range: percent
  - Mean Reciprocal Rank at rank 20. Calculated as the average of 1/rank for each session, where rank is the position of the ground-truth item in the recommended list (capped at 20).

## Input / output format

**Input**: A sequence of items representing a user's session history, typically ordered by timestamp.

**Output**: A ranked list of candidate items for the next interaction.

## Scoring recipe

```python
def compute_metrics(recommended_list, ground_truth, k=20):
    hits = 1 if ground_truth in recommended_list[:k] else 0
    precision = hits / k
    rank = recommended_list.index(ground_truth) + 1 if ground_truth in recommended_list else k + 1
    mrr = 1 / rank if rank <= k else 0
    return precision, mrr
```

## Common pitfalls

- Ignoring the 'timeframe' field to sort session items sequentially degrades performance and causes score discrepancies with prior work.
- The benchmark reports P@20 rather than the more common P@10 or P@5, which can mislead comparisons if not noted.

## Evidence (verbatim from paper)

> The scores on Diginetical dataset differ from results reported in [9,11] because they did not sort the session items according to "timeframe" field, which ignores the sequential information. CSRM outperforms them under the P@20 metric on LastFM dataset. Fig. 3 shows results against the hyper-parameter k on three datasets. The left Y-axis refers to P@20 (%) and the right Y-axis refers to MRR@20 (%).

## Citation

```bibtex
@misc{dou2021pen4rec,
  title={PEN4Rec: Preference Evolution Networks for Session-based Recommendation},
  author={Dou Hu et al.},
  year={2021},
  note={arXiv:2106.09306}
}
```

- arXiv: 2106.09306