Criteo Dlrm Eval

---
name: criteo-dlrm-eval
description: Evaluates the predictive accuracy and inference efficiency of deep learning recommendation models (DLRM) with compressed embedding tables on large-scale advertising click-through rate datasets. It measures Area Under the ROC Curve (AUC) to assess model quality and samples per second to quantify inference throughput under memory-constrained conditions. Use when the user wants to benchmark on CriteoTB, Criteo Kaggle, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.02191
  bibtex_key: desai2021robe
  confidence: high
---

# criteo-dlrm-eval

> Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference — Desai et al. (2021) (arXiv:2108.02191, 2021)

## What this evaluates

Evaluates the predictive accuracy and inference efficiency of deep learning recommendation models (DLRM) with compressed embedding tables on large-scale advertising click-through rate datasets. It measures Area Under the ROC Curve (AUC) to assess model quality and samples per second to quantify inference throughput under memory-constrained conditions.

## Datasets

- **CriteoTB** — total ?; splits: train (-1), val (-1), test (-1)
- **Criteo Kaggle** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Computed over predicted click-through rate probabilities and binary ground-truth labels.
- `Throughput` — range: other
  - Inference speed measured in samples processed per second. Includes data transfer from RAM to GPU and hash computation.

## Input / output format

**Input**: Batch of 13 integer features and 26 categorical features (tokens) representing user/ad interactions.

**Output**: Binary click/no-click label (0 or 1) or predicted probability score.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    # y_true: binary ground truth labels
    # y_pred: predicted probabilities
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)
```

## Common pitfalls

- The MLPerf CriteoTB benchmark strictly requires achieving an AUC of at least 0.8025 to be considered valid; simply reporting AUC without meeting this threshold may not be comparable.
- Inference throughput measurements include data transfer from RAM to GPU and hash computation, not just the forward pass, which can skew comparisons if not standardized.
- Training time comparisons are confounded by the number of epochs required to reach target AUC, as compressed models often need more epochs despite smaller memory footprint.

## Evidence (verbatim from paper)

> We will use the same quality metric of 0.8025 AUC as prescribed in MLPerf settings for CriteoTB dataset to evaluate ROBE-Z.

## Citation

```bibtex
@misc{desai2021robe,
  title={Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference},
  author={Desai et al. (2021)},
  year={2021},
  note={arXiv:2108.02191}
}
```

- arXiv: 2108.02191