Criteo Ctr Eval

---
name: criteo-ctr-eval
description: Evaluates the predictive quality and system efficiency of deep learning recommendation models on click-through rate prediction. It measures how well parameter-sharing compression techniques maintain model accuracy while reducing memory footprint and improving training and inference latency. Use when the user wants to benchmark on criteo-kaggle, criteo-tb, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.10731
  bibtex_key: desai2022modelsizecompression
  confidence: high
---

# criteo-ctr-eval

> The trade-offs of model size in large recommendation models : A 10000 $\times$ compressed criteo-tb DLRM model (100 GB parameters to mere 10MB) — Desai et al. (2022) (arXiv:2207.10731, 2022)

## What this evaluates

Evaluates the predictive quality and system efficiency of deep learning recommendation models on click-through rate prediction. It measures how well parameter-sharing compression techniques maintain model accuracy while reducing memory footprint and improving training and inference latency.

## Datasets

- **criteo-kaggle** — total ?; splits: train (-1), val (-1), test (-1)
- **criteo-tb** — total 800000000; splits: test (89000000)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, computed over predicted click probabilities and binary ground truth labels to measure ranking quality.

## Input / output format

**Input**: 13 integer features and 26 categorical features per sample.

**Output**: Binary classification probability (click/no-click) for CTR prediction.

## Scoring recipe

```python
def compute_auc(predictions, labels):
    # predictions: array of float probabilities
    # labels: array of binary ground truth labels
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)
```

## Common pitfalls

- Assuming high compression ratios (e.g., 10000x) inherently degrade model quality; the protocol shows AUC remains stable up to extreme compression.
- Focusing only on convergence speed (epochs) without accounting for per-iteration latency gains; overall training time remains comparable due to faster computation per iteration.

## Evidence (verbatim from paper)

> Facebook MLPerf DLRM[2] model, available under Apache-2.0 license, for the criteo-tb dataset achieves the target AUC (0.8025) with the embedding memory of around 100GB. This model uses a maximum cap of 40M indices per embedding table, leading to a total of 204M embeddings.

## Citation

```bibtex
@misc{desai2022modelsizecompression,
  title={The trade-offs of model size in large recommendation models : A 10000 $\times$ compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)},
  author={Desai et al. (2022)},
  year={2022},
  note={arXiv:2207.10731}
}
```

- arXiv: 2207.10731