Casif Eval

---
name: casif-eval
description: This evaluation protocol assesses a model's ability to perform session-based next-item recommendation by predicting the subsequent item a user will click based on their recent interaction history. It probes the model's capacity to capture both short-term sequential dependencies and long-term contextual patterns within a session without relying on explicit user profiles. Use when the user wants to benchmark on Yoochoose1/64, Yoochoose1/4, Diginetica, or asks about evaluating this task. Reports Recall@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.15514
  bibtex_key: duan2021casif
  confidence: high
---

# casif-eval

> Context-aware short-term interest first model for session-based recommendation — Duan et al. (2021) (arXiv:2103.15514, 2021)

## What this evaluates

This evaluation protocol assesses a model's ability to perform session-based next-item recommendation by predicting the subsequent item a user will click based on their recent interaction history. It probes the model's capacity to capture both short-term sequential dependencies and long-term contextual patterns within a session without relying on explicit user profiles.

## Datasets

- **Yoochoose1/64** — total 424757; splits: train (368859), test (55898)
- **Yoochoose1/4** — total 5973643; splits: train (5917745), test (55898)
- **Diginetica** — total 780328; splits: train (719470), test (60858)

## Metrics

- `Recall@k` **(primary)** — range: percent
  - Recall@k = n_hit / N, where N is the number of test sessions and n_hit is the number of sessions where the ground-truth next item appears in the top-K predicted items.
- `MRR@k` — range: percent
  - MRR@k = (1/N) * sum(1/rank_i), where rank_i is the position of the ground-truth item in the top-K list (0 if not present).

## Input / output format

**Input**: Chronological sequence of items clicked in a session.

**Output**: Ranked list of candidate items (top-K) for the next click.

## Scoring recipe

```python
def compute_recall_at_k(preds, gold, k):
    hits = sum(1 for p, g in zip(preds, gold) if g in p[:k])
    return hits / len(gold)

def compute_mrr_at_k(preds, gold, k):
    rr = []
    for p, g in zip(preds, gold):
        try:
            rr.append(1.0 / (p[:k].index(g) + 1))
        except ValueError:
            rr.append(0.0)
    return sum(rr) / len(gold)
```

## Common pitfalls

- The standard preprocessing for Yoochoose and Diginetica involves filtering sessions of length 1 and items with fewer than 5 occurrences, plus specific time-based train/test splits; using raw data yields non-comparable results.
- Metrics are reported as percentages in the results tables but defined as proportions in the formulas; failing to multiply by 100 causes a 100x discrepancy.
- Evaluation must be performed at multiple K values (5, 10, 20); reporting only one K value omits critical performance dimensions highlighted in the paper.

## Evidence (verbatim from paper)

> We use the following performance metrics to compare these algorithms, which have been widely used in session-based recommendation systems.

Recall@k: Be widely used as a measure of predictive accuracy in all kinds of recommendation systems. It represents the proportion of correctly recommended items amongst the top-k items.

$$
\text {R e c a l l} @ k = \frac {n _ {\text {h i t}}}{N}, \tag {14}
$$

Where  $N$  is the number of test sessions in the testing set,  $n_{hit}$ denotes the number of sessions which have hit items among top-K ranking list.

$MRR@k$ : MRR (Mean Reciprocal Rank) is the average of reciprocal ranks of desired items. The reciprocal rank is set to zero if the rank is larger than  $K$ .

$$
M R R @ k = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{r a n k _ {i}}, \tag {15}
$$

## Citation

```bibtex
@misc{duan2021casif,
  title={Context-aware short-term interest first model for session-based recommendation},
  author={Duan et al. (2021)},
  year={2021},
  note={arXiv:2103.15514}
}
```

- arXiv: 2103.15514