Tenrec Eval

---
name: tenrec-eval
description: Evaluates recommender systems across multiple tasks including click-through rate (CTR) prediction, sequential recommendation, and top-N item ranking. It probes cross-domain generalization, cold-start handling, and the sensitivity of ranking metrics to negative sampling strategies. Use when the user wants to benchmark on Tenrec, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.10629
  bibtex_key: yuan2022tenrec
  confidence: high
---

# tenrec-eval

> Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems — Yuan et al. (2022) (NeurIPS 2022, 2022)

## What this evaluates

Evaluates recommender systems across multiple tasks including click-through rate (CTR) prediction, sequential recommendation, and top-N item ranking. It probes cross-domain generalization, cold-start handling, and the sensitivity of ranking metrics to negative sampling strategies.

## Datasets

- **Tenrec** — total 5000000; splits: train (-1), val (-1), test (-1); repo https://github.com/yuangh-x/2022-NIPS-Tenrec

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. It measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `Logloss` — range: other
  - Binary cross-entropy loss: -1/N * sum(y*log(p) + (1-y)*log(1-p)). Measures the performance of a probabilistic classifier.
- `NDCG@20` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 20. Compares the predicted ranking to the ground truth, applying a logarithmic discount to positions.
- `HR@20` — range: [0, 1]
  - Hit Rate at rank 20. Proportion of test items that appear in the top-20 recommended items.
- `Recall@20` — range: [0, 1]
  - Proportion of relevant items in the ground truth that are retrieved in the top-20 recommendations.

## Input / output format

**Input**: User and item features, interaction history (sequence of items), and contextual metadata. For CTR: user/item features and context. For Rec: sequence of previously interacted items.

**Output**: CTR: probability of click. Rec: ranked list of candidate items.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    pos = y_pred[y_true == 1]
    neg = y_pred[y_true == 0]
    concordant = sum(p > n for p in pos for n in neg)
    return concordant / (len(pos) * len(neg))

def compute_ndcg_at_k(y_true, y_pred, k=20):
    ranked = np.argsort(y_pred)[::-1][:k]
    dcg = sum((2**y_true[i] - 1) / np.log2(i + 2) for i in ranked)
    ideal = sorted(y_true, reverse=True)[:k]
    idcg = sum((2**i - 1) / np.log2(j + 2) for j, i in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0
```

## Common pitfalls

- Negative sampling strategy heavily influences results; using advanced samplers (e.g., dynamic or popularity-based) can artificially inflate accuracy compared to random sampling.
- Cold-start evaluation must strictly isolate cold users and use only their held-out interactions (25% val, 25% test) to avoid data leakage from warm users.
- Standard top-N evaluation filters out users with session length < 10, which significantly changes the user population compared to unfiltered splits.

## Evidence (verbatim from paper)

> Table 13: Results for CTR prediction.

<table border=1 style='margin: auto; width: max-content;'><tr><td style='text-align: center;'>Model</td><td style='text-align: center;'>AUC</td><td style='text-align: center;'>Logloss</td></tr>

## Citation

```bibtex
@misc{yuan2022tenrec,
  title={Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems},
  author={Yuan et al. (2022)},
  year={2022},
  note={NeurIPS 2022}
}
```

- arXiv: 2210.10629