Cape Sr Eval

---
name: cape-sr-eval
description: This evaluation protocol assesses the effectiveness of context-aware position encoding methods in sequential recommendation systems. It measures how well models rank a target item given a user's historical interaction sequence, testing the model's ability to capture temporal and semantic dependencies in user behavior. Use when the user wants to benchmark on AmazonElectronics, KuaiVideo, AmazonBooks, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.09027
  bibtex_key: yuan2025cape
  confidence: high
---

# cape-sr-eval

> A Contextual-Aware Position Encoding for Sequential Recommendation — Yuan et al. (2025) (arXiv:2502.09027, 2025)

## What this evaluates

This evaluation protocol assesses the effectiveness of context-aware position encoding methods in sequential recommendation systems. It measures how well models rank a target item given a user's historical interaction sequence, testing the model's ability to capture temporal and semantic dependencies in user behavior.

## Datasets

- **AmazonElectronics** — total 1689188; splits: train (-1), val (-1), test (-1); HF `reczoo/AmazonElectronics`
- **KuaiVideo** — total 3239534; splits: train (-1), val (-1), test (-1); HF `reczoo/KuaiVideo`
- **AmazonBooks** — total 1689188; splits: train (-1), val (-1), test (-1); HF `reczoo/AmazonBooks`

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Probability that a randomly chosen positive sample is ranked higher than a randomly chosen negative sample.
- `gAUC` — range: [0, 1]
  - AUC computed per user separately, then averaged with a weighted average across users.
- `Logloss` — range: [0, ∞)
  - Negative log-likelihood of the predicted probability for the true label.
- `Recall@K` — range: [0, 1]
  - Fraction of relevant items retrieved in the top-K recommendations.
- `NDCG@K` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank K, measuring ranking quality with position decay.

## Input / output format

**Input**: User interaction history represented as sequences of item IDs and category IDs, along with a target item ID to predict. Context length is capped at 100 (or 50 for AmazonBooks).

**Output**: A continuous prediction score (probability) for the target item, used for ranking against negative samples.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    pos = y_pred[y_true == 1]
    neg = y_pred[y_true == 0]
    if len(pos) == 0 or len(neg) == 0: return 0.0
    return (pos[:, None] > neg[None, :]).mean()

def compute_gauc(y_true, y_pred, user_ids):
    user_auc, weights = [], []
    for uid in np.unique(user_ids):
        mask = user_ids == uid
        if mask.sum() < 2: continue
        user_auc.append(compute_auc(y_true[mask], y_pred[mask]))
        weights.append(mask.sum())
    return np.average(user_auc, weights=weights)
```

## Common pitfalls

- Applying standard NLP position encodings (e.g., RoPE, CoPE) directly to sequential recommendation often degrades performance, especially on short contexts (<100 items).
- AUC/gAUC improvements on the third decimal place are reported as statistically significant, which is unusually high precision for recommendation benchmarks and may mask practical relevance.
- Metrics are averaged over 5 random seeds, but the paper does not specify the exact train/val/test split ratios or temporal ordering conventions used for the reczoo datasets.

## Evidence (verbatim from paper)

> We evaluate performance using gAUC (Zhou et al., [2018]), AUC (Guo et al., [2017]) and Logloss. AUC measures the probability of a positive sample being ranked higher than a randomly chosen negative one. gAUC is to compute the AUC for each user separately, and then make a weighted average for them. In the experiment with large scale model, the metrics are Recall@K (R@K) and NDCG@K (N@K).

## Citation

```bibtex
@misc{yuan2025cape,
  title={A Contextual-Aware Position Encoding for Sequential Recommendation},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2502.09027}
}
```

- arXiv: 2502.09027