Share Eval

---
name: share-eval
description: Evaluates a model's ability to predict the next item in an anonymous user session based on sequential click history. It probes the model's capacity to capture short-term user intent and higher-order item correlations within dynamic session contexts. Use when the user wants to benchmark on YooChoose, Diginetica, or asks about evaluating this task. Reports Hit@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.14266
  bibtex_key: wang2021share
  confidence: high
---

# share-eval

> Session-based Recommendation with Hypergraph Attention Networks — Wang et al. (2021) (arXiv:2112.14266, 2021)

## What this evaluates

Evaluates a model's ability to predict the next item in an anonymous user session based on sequential click history. It probes the model's capacity to capture short-term user intent and higher-order item correlations within dynamic session contexts.

## Datasets

- **YooChoose** — total ?; splits: train (-1), test (-1)
- **Diginetica** — total ?; splits: train (-1), test (-1)

## Metrics

- `Hit@20` **(primary)** — range: percent
  - Binary indicator: 1 if the ground-truth item's rank is ≤ 20, else 0. Averaged over all test sessions.
- `MRR@20` — range: percent
  - Reciprocal rank: 1/r_s if ground-truth rank r_s ≤ 20, else 0. Averaged over all test sessions.

## Input / output format

**Input**: A sequence of items clicked by a user within a single anonymous session.

**Output**: A ranked list of candidate items for the next click, typically truncated to top-K predictions.

## Scoring recipe

```python
def evaluate(predictions, ground_truths, K=20):
    hits = 0
    mrrs = 0.0
    for pred_list, gt in zip(predictions, ground_truths):
        rank = pred_list.index(gt) + 1 if gt in pred_list else K + 1
        if rank <= K:
            hits += 1
            mrrs += 1.0 / rank
    return hits / len(predictions) * 100, mrrs / len(predictions) * 100
```

## Common pitfalls

- Leave-one-out evaluation: only the last item in each session is held out as ground truth for testing; all prior items are used as context.
- Evaluation is restricted to the item vocabulary present in the training set; unseen items are excluded from the candidate list.
- K=20 is the standard reporting threshold for overall comparison, while ablation studies also report K=10.
- Statistical significance of improvements is verified using a paired t-test with p < 0.05.

## Evidence (verbatim from paper)

> As in previous works for session-based recommendation [16, 21, 32], we adopt both Mean Reciprocal Rank (MRR@K) and Hit Rate (Hit@K) as evaluation metrics. Given the ranked list of items predicted for each session, Hit@K measures the probability that the ground-truth item is within the top-K. Let r_s denote the ranking of the ground-truth item for session s. Then, Hit_s@K = 1 if r_s <= K and Hit_s@K = 0 otherwise. As for MRR@K, it measures the average ranking of the ground-truth items among the lists. That is MRR_s@K = 1/r_s if r_s <= K otherwise MRR_s@K = 0. Then we take the average values of MRR and Hit Rate over all the sessions in the test set and report the results.

## Citation

```bibtex
@misc{wang2021share,
  title={Session-based Recommendation with Hypergraph Attention Networks},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.14266}
}
```

- arXiv: 2112.14266