Rec Auc Eval

---
name: rec-auc-eval
description: Evaluates the predictive performance of recommendation models on large-scale click-through rate datasets. It specifically probes how model scalability and embedding size affect ranking quality, revealing the phenomenon of embedding collapse when scaling up feature interactions. Use when the user wants to benchmark on Criteo, Avazu, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.04400
  bibtex_key: guo2023embeddingcollapse
  confidence: high
---

# rec-auc-eval

> On the Embedding Collapse when Scaling up Recommendation Models — Guo et al. (2023) (arXiv:2310.04400, 2023)

## What this evaluates

Evaluates the predictive performance of recommendation models on large-scale click-through rate datasets. It specifically probes how model scalability and embedding size affect ranking quality, revealing the phenomenon of embedding collapse when scaling up feature interactions.

## Datasets

- **Criteo** — total 45800000; splits: train (36640000), val (4580000), test (4580000)
- **Avazu** — total 40400000; splits: train (32320000), val (4040000), test (4040000)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (click) is ranked higher than a randomly chosen negative instance (non-click).

## Input / output format

**Input**: Tabular data with categorical fields representing user and item features for click-through rate prediction.

**Output**: Probability score indicating the likelihood of a click.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)
```

## Common pitfalls

- Scaling embedding dimensions (2x to 10x) does not improve AUC due to embedding collapse; performance plateaus or degrades.
- Early stopping is applied on validation AUC with a patience of 3, not on training loss or test performance.
- Experiments are repeated 3 times with different random initializations, but dataset splits use a fixed random seed (0).

## Evidence (verbatim from paper)

> For all experiments, we split the dataset into 8:1:1 for training/validation/test with random seed 0. ... All experiments use early stopping on validation AUC with patience 3. We repeat each experiment for 3 times with different random initialization.

## Citation

```bibtex
@misc{guo2023embeddingcollapse,
  title={On the Embedding Collapse when Scaling up Recommendation Models},
  author={Guo et al. (2023)},
  year={2023},
  note={arXiv:2310.04400}
}
```

- arXiv: 2310.04400