Deepfm Ctr Eval

---
name: deepfm-ctr-eval
description: Evaluates click-through rate (CTR) prediction models by measuring their ability to correctly rank clicked versus non-clicked instances and output calibrated click probabilities. Use when the user wants to benchmark on Criteo Dataset, Company* Dataset, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1703.04247
  bibtex_key: guo2017deepfm
  confidence: high
---

# deepfm-ctr-eval

> DeepFM: A Factorization-Machine based Neural Network for CTR Prediction — Guo et al. (2017) (arXiv:1703.04247, 2017)

## What this evaluates

Evaluates click-through rate (CTR) prediction models by measuring their ability to correctly rank clicked versus non-clicked instances and output calibrated click probabilities.

## Datasets

- **Criteo Dataset** — total 45000000; splits: train (-1), test (-1)
- **Company* Dataset** — total 1000000000; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `Logloss` — range: other
  - Binary cross-entropy loss: -mean(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred)). Lower values indicate better probability calibration.

## Input / output format

**Input**: A feature vector containing 13 continuous features and 26 categorical features (or app/user/context features for Company*), paired with a binary label indicating whether the user clicked.

**Output**: A single scalar probability score representing the predicted likelihood of a click.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    pos = y_true == 1
    neg = y_true == 0
    return np.mean(y_pred[pos][:, None] > y_pred[neg][None, :])

def compute_logloss(y_true, y_pred):
    eps = 1e-15
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
```

## Common pitfalls

- The Criteo dataset uses a random 90/10 split rather than a temporal split, which can cause data leakage if the model is evaluated on future data in practice.
- Logloss is reported to 5 decimal places, but absolute differences of ~0.0001 may lack statistical significance without variance or confidence intervals.
- Efficiency is measured as a relative ratio to LR training time, not absolute wall-clock seconds, making cross-hardware comparisons unreliable.

## Evidence (verbatim from paper)

> We use two evaluation metrics in our experiments: AUC (Area Under ROC) and Logloss (cross entropy). The performance for CTR prediction of different models on Criteo dataset and Company* dataset is shown in Table 2, where we have the following observations: DeepFM outperforms LR by 0.86% and 4.18% in terms of AUC (1.15% and 5.60% in terms of Logloss) on Company* and Criteo datasets.

## Citation

```bibtex
@misc{guo2017deepfm,
  title={DeepFM: A Factorization-Machine based Neural Network for CTR Prediction},
  author={Guo et al. (2017)},
  year={2017},
  note={arXiv:1703.04247}
}
```

- arXiv: 1703.04247