Sten Social Temporal Rec Eval

---
name: sten-social-temporal-rec-eval
description: Evaluates sequential recommendation models that incorporate social influence and temporal dynamics. It probes the ability to predict the next item a user will interact with based on their historical behavior sequence, social connections, and event timestamps. Use when the user wants to benchmark on Delicious, Yelp, Ciao, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.13539
  bibtex_key: li2021extracting
  confidence: high
---

# sten-social-temporal-rec-eval

> Extracting Attentive Social Temporal Excitation for Sequential Recommendation — Li et al. (2021) (arXiv:2109.13539, 2021)

## What this evaluates

Evaluates sequential recommendation models that incorporate social influence and temporal dynamics. It probes the ability to predict the next item a user will interact with based on their historical behavior sequence, social connections, and event timestamps.

## Datasets

- **Delicious** — total ?; splits: train (-1), val (-1), test (-1); repo https://grouplens.org/datasets/hetrec-2011/
- **Yelp** — total ?; splits: train (-1), val (-1), test (-1); repo https://www.yelp.com/dataset
- **Ciao** — total ?; splits: train (-1), val (-1), test (-1); repo https://www.cse.msu.edu/tangjili/datasetcode/truststudy.htm

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Percentage of ground-truth relevant items appearing within the top-10 ranked list. Equivalent to hit ratio for next-item recommendation.
- `Recall@20` — range: [0, 1]
  - Percentage of ground-truth relevant items appearing within the top-20 ranked list.
- `NDCG@10` — range: [0, 1]
  - Standard ranking metric reflecting correlation and position. For next-item recommendation, formulated as NDCG = 1/log2(1+r_p), where r_p is the rank of the positive item.
- `NDCG@20` — range: [0, 1]
  - Same as NDCG@10 but evaluated over the top-20 ranked list.
- `MRR@10` — range: [0, 1]
  - Mean reciprocal rank of the target item. Calculated as the mean of the reciprocal of the target item's actual rank. If rank > 10, reciprocal rank is set to zero.
- `MRR@20` — range: [0, 1]
  - Same as MRR@10 but truncates reciprocal rank to zero only if rank > 20.

## Input / output format

**Input**: User interaction history (sequence of item IDs with timestamps), social graph (friend lists/relations), and implicit feedback labels (1 for observed interactions).

**Output**: A ranked list of candidate items for the next interaction.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k):
    # predictions: list of recommended item IDs
    # ground_truth: single target item ID
    hit = 1 if ground_truth in predictions[:k] else 0
    recall = hit
    
    if ground_truth in predictions:
        rank = predictions.index(ground_truth) + 1
        dcg = 1.0 / math.log2(1 + rank)
        ndcg = dcg
        mrr = 1.0 / rank if rank <= k else 0.0
    else:
        ndcg = 0.0
        mrr = 0.0
    return recall, ndcg, mrr
```

## Common pitfalls

- Yelp dataset uses only the first 10% of data chronologically for experiments, unlike standard full-dataset splits.
- MRR@k truncates the reciprocal rank to 0 if the target item's rank exceeds k, deviating from standard MRR which averages over all ranks.
- NDCG formula provided is simplified for a single positive item per query, ignoring standard DCG normalization over top-k positions.
- All ratings/reviews are binarized to implicit feedback (1), discarding explicit rating magnitudes and requiring careful handling of negative samples.

## Evidence (verbatim from paper)

> We adopt several common evaluation metrics to evaluate the recommendation performance, including Recall, Normalized Discount Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR): Recall@k. It means the percentage of groundtruth relevant items appear within the top-k ranking list. Consider to next-item recommendation, it is equivalent to the hit ratio. NDCG@k. It is a standard ranking metric and reflects both the correlation and position for each recommended item. For next-item recommendation, it is formulated as NDCG = 1/log2(1+r_p), where r_p is the rank of the positive item. MRR@k. It takes the actual rank of target item into consideration and is calculated by the mean value of the reciprocal of target item’s actual rank. When the actual rank is out of k, the reciprocal rank is set to zero.

## Citation

```bibtex
@misc{li2021extracting,
  title={Extracting Attentive Social Temporal Excitation for Sequential Recommendation},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2109.13539}
}
```

- arXiv: 2109.13539