Sru Eval

---
name: sru-eval
description: Evaluates the recommendation accuracy and unlearning effectiveness of session-based recommendation models after deleting a portion of training sessions. It measures how well the model retains predictive performance while successfully preventing the inference of removed items. Use when the user wants to benchmark on Amazon Beauty, Amazon Games, Steam, or asks about evaluating this task. Reports NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.14447
  bibtex_key: xin2023sru
  confidence: high
---

# sru-eval

> On the Effectiveness of Unlearning in Session-Based Recommendation — Xin et al. (2023) (arXiv:2312.14447, 2023)

## What this evaluates

Evaluates the recommendation accuracy and unlearning effectiveness of session-based recommendation models after deleting a portion of training sessions. It measures how well the model retains predictive performance while successfully preventing the inference of removed items.

## Datasets

- **Amazon Beauty** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon Games** — total ?; splits: train (-1), val (-1), test (-1)
- **Steam** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `NDCG@K` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K. Assigns higher scores to top-ranked positions. Computed as sum_{i=1}^K (rel_i / log2(i+1)) / ideal_NDCG@K.
- `Recall@K` — range: [0, 1]
  - Binary indicator: 1 if the ground-truth item appears in the top-K recommended items, 0 otherwise.
- `HIT@K` — range: [0, 1]
  - Probability that an unlearned item can be inferred or reconstructed from the remaining interactions. Lower scores indicate better unlearning effectiveness.

## Input / output format

**Input**: Sequence of the last 10 interacted items (Beauty) or last 20 interacted items (Games, Steam), padded with a padding token if shorter.

**Output**: Ranked list of items from the entire item vocabulary.

## Scoring recipe

```python
def compute_metrics(preds, gold, k_vals=[10, 20]):
    scores = {}
    for k in k_vals:
        top_k = preds[:k]
        recall = 1.0 if gold in top_k else 0.0
        ndcg = 0.0
        for i, item in enumerate(top_k):
            if item == gold:
                ndcg = 1.0 / math.log2(i + 2)
                break
        scores[f'Recall@{k}'] = recall
        scores[f'NDCG@{k}'] = ndcg
    return scores
```

## Common pitfalls

- HIT@K measures unlearning success, so lower scores indicate better performance (counter-intuitive for standard recommendation metrics).
- Exact unlearning is inherently impossible in session-based recommendation due to sequential and collaborative dependencies; metrics reflect residual inference probability rather than exact deletion.

## Evidence (verbatim from paper)

> To evaluate recommendation performance, we adopt two common top-K metrics: Recall@K and NDCG@K. Recall@K measures whether the ground-truth item is in the top-K positions of the recommendation list [38]. NDCG@K is a weighted metric that assigns higher scores to top-ranked positions [19]. We use the metric HIT described in section 4.4 to evaluate unlearning effectiveness.

## Citation

```bibtex
@misc{xin2023sru,
  title={On the Effectiveness of Unlearning in Session-Based Recommendation},
  author={Xin et al. (2023)},
  year={2023},
  note={arXiv:2312.14447}
}
```

- arXiv: 2312.14447