Caser Sequential Rec Eval

---
name: caser-sequential-rec-eval
description: Evaluates a model's ability to capture sequential user behavior patterns for personalized top-N item recommendation. It probes the model's capacity to model temporal dependencies, skip behaviors, and union-level sequential patterns from historical interactions to predict future items. Use when the user wants to benchmark on MovieLens, Gowalla, Foursquare, Tmall, or asks about evaluating this task. Reports MAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.07426
  bibtex_key: tang2018caser
  confidence: high
---

# caser-sequential-rec-eval

> Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding — Tang et al. (2018) (arXiv:1809.07426, 2018)

## What this evaluates

Evaluates a model's ability to capture sequential user behavior patterns for personalized top-N item recommendation. It probes the model's capacity to model temporal dependencies, skip behaviors, and union-level sequential patterns from historical interactions to predict future items.

## Datasets

- **MovieLens** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
- **Gowalla** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
- **Foursquare** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
- **Tmall** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser

## Metrics

- `Precision@N` — range: [0, 1]
  - Precision@N = |R ∩ R̂_1:N| / N, where R is the test set (last 20% of user's sequence) and R̂_1:N is the top-N predicted items. Averaged over all users.
- `Recall@N` — range: [0, 1]
  - Recall@N = |R ∩ R̂_1:N| / |R|, where R is the test set and R̂_1:N is the top-N predicted items. Averaged over all users.
- `MAP` **(primary)** — range: [0, 1]
  - Mean Average Precision. AP = (Σ_{N=1}^{|R̂|} Prec@N × rel(N)) / |R̂|, where rel(N)=1 if the N-th predicted item is in R. MAP is the average AP across all users.

## Input / output format

**Input**: A user's chronological sequence of past item interactions, converted to implicit feedback (binary 1). The sequence is processed up to a maximum length L (Markov order).

**Output**: A ranked list of top-N predicted items for the user.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, Ns=[1, 5, 10]):
    precisions, recalls, aps = [], [], []
    for pred, gold in zip(predictions, ground_truth):
        gold_set = set(gold)
        for N in Ns:
            top_n = set(pred[:N])
            precisions.append(len(top_n & gold_set) / N)
            recalls.append(len(top_n & gold_set) / len(gold))
        ap_sum = 0.0
        for k, item in enumerate(pred):
            if item in gold_set:
                ap_sum += len(set(pred[:k+1]) & gold_set) / (k + 1)
        aps.append(ap_sum / len(pred))
    return {
        'Precision@N': {N: np.mean([p for p in precisions]) for N in Ns},
        'Recall@N': {N: np.mean([r for r in recalls]) for N in Ns},
        'MAP': np.mean(aps)
    }
```

## Common pitfalls

- Data is split chronologically per user (70% train, 10% val, 20% test), not randomly shuffled.
- Cold-start users and items are filtered out (minimum 5-15 interactions depending on dataset).
- All numeric ratings are binarized to implicit feedback (1), discarding rating magnitude.
- GRU4Rec is session-based (no user ID), making direct comparison to personalized models potentially unfair for long-term preference modeling.

## Evidence (verbatim from paper)

> As in [19, 21, 29, 32], we evaluate a model by Precision@N, Recall@N, and Mean Average Precision (MAP). Given a list of top N predicted items for a user, denoted R̂_1:N, and the last 20% of actions in her/his sequence (i.e., denoted R (i.e., the test set), Precision@N and Recall@N are computed by Prec@N = |R ∩ R̂_1:N| / N, Recall@N = |R ∩ R̂_1:N| / |R|. We report the average of these values of all users. N∈{1,5,10} The Average Precision (AP) is defined by AP = (sum_{N=1}^{|R̂|} Prec@N × rel(N)) / |R̂|, where rel(N) = 1 if the N-th item in R̂ is in R. The Mean Average Precision (MAP) is the average of AP for all users.

## Citation

```bibtex
@misc{tang2018caser,
  title={Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding},
  author={Tang et al. (2018)},
  year={2018},
  note={arXiv:1809.07426}
}
```

- arXiv: 1809.07426