Kstt Eval

---
name: kstt-eval
description: Evaluates session-based recommendation models on predicting the next item in a user's click sequence by integrating knowledge graph attributes and temporal dynamics between clicks. Use when the user wants to benchmark on Yoochoose, Diginetica, Last-fm, or asks about evaluating this task. Reports Recall@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.08745
  bibtex_key: zhang2021knowledgeenhanced
  confidence: high
---

# kstt-eval

> Knowledge-enhanced Session-based Recommendation with Temporal Transformer — Zhang et al. (2021) (arXiv:2112.08745, 2021)

## What this evaluates

Evaluates session-based recommendation models on predicting the next item in a user's click sequence by integrating knowledge graph attributes and temporal dynamics between clicks.

## Datasets

- **Yoochoose** — total ?; splits: Yoochoose1/64 (-1), Yoochoose1/4 (-1)
- **Diginetica** — total ?; splits: test (-1)
- **Last-fm** — total ?; splits: test (-1)

## Metrics

- `Recall@20` **(primary)** — range: [0, 1]
  - Fraction of test sessions where the ground-truth next item appears in the top-20 recommended items.
- `MRR@20` — range: [0, 1]
  - Average of the reciprocal ranks of the first correct item within the top-20 recommendations across all test sessions.

## Input / output format

**Input**: Session sequences of item IDs, augmented with item attributes from a knowledge graph and time intervals between consecutive clicks.

**Output**: Ranked list of top-20 candidate items for the next click.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    recalls, mrrs = [], []
    for pred, g in zip(predictions, gold):
        if g in pred:
            recalls.append(1.0)
            mrrs.append(1.0 / (pred.index(g) + 1))
        else:
            recalls.append(0.0)
            mrrs.append(0.0)
    return sum(recalls) / len(recalls), sum(mrrs) / len(mrrs)
```

## Common pitfalls

- Datasets use fixed temporal cutoffs for train/test splits, so random shuffling will cause data leakage.
- K is fixed at 20 in the reported tables, but the protocol supports arbitrary K.
- Last-fm requires strict filtering: keep top 40,000 artists and sessions with length between 2 and 50.

## Evidence (verbatim from paper)

> For the evaluation metrics, we use R@K (Recall@K) and MRR@K (Mean Reciprocal Rank@K) to measure the recommendation performance.

## Citation

```bibtex
@misc{zhang2021knowledgeenhanced,
  title={Knowledge-enhanced Session-based Recommendation with Temporal Transformer},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2112.08745}
}
```

- arXiv: 2112.08745