Ctr Prediction Eval

---
name: ctr-prediction-eval
description: Evaluates the ability of deep learning models to predict click-through rates (CTR) from sparse, high-dimensional categorical features in advertising and recommendation scenarios. It probes how well models capture multi-scale semantic interactions and handle large-scale, imbalanced binary classification tasks typical of real-world ad systems. Use when the user wants to benchmark on Avazu, MovieLens, Weibo, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1812.01353
  bibtex_key: niu2018structured
  confidence: high
---

# ctr-prediction-eval

> Structured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction — Niu et al. (2018) (arXiv:1812.01353, 2018)

## What this evaluates

Evaluates the ability of deep learning models to predict click-through rates (CTR) from sparse, high-dimensional categorical features in advertising and recommendation scenarios. It probes how well models capture multi-scale semantic interactions and handle large-scale, imbalanced binary classification tasks typical of real-world ad systems.

## Datasets

- **Avazu** — total 40000000; splits: train (-1), test (-1)
- **MovieLens** — total 20000263; splits: train (-1), test (-1)
- **Weibo** — total 1100000000; splits: train (1000000000), test (100000000)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the ranking quality of predicted CTR scores against binary click/no-click labels across all intra-user and inter-user pairs.
- `RelaImpr` — range: percent
  - Relative improvement over a baseline model, calculated as ((AUC_model - 0.5) / (AUC_base - 0.5) - 1) * 100%. Used to normalize performance gains across datasets with different difficulty levels.

## Input / output format

**Input**: Sparse categorical feature vectors (22–31 fields covering user profile, ad information, and context) with a binary click/no-click label.

**Output**: A single continuous probability score (predicted CTR) per sample.

## Scoring recipe

```python
def compute_auc(labels, preds):
    total_pos = sum(labels)
    total_neg = len(labels) - total_pos
    sorted_idx = np.argsort(preds)[::-1]
    labels_sorted = np.array(labels)[sorted_idx]
    tp = fp = 0
    auc = 0.0
    prev_fpr = prev_tpr = 0.0
    for y in labels_sorted:
        if y == 1: tp += 1
        else: fp += 1
        tpr = tp / total_pos
        fpr = fp / total_neg
        auc += (fpr - prev_fpr) * (tpr + prev_tpr) / 2
        prev_fpr, prev_tpr = fpr, tpr
    return auc

def compute_rela_impr(auc_model, auc_base):
    return ((auc_model - 0.5) / (auc_base - 0.5) - 1) * 100
```

## Common pitfalls

- RelaImpr becomes numerically unstable or undefined if the baseline AUC is close to 0.5, as the denominator approaches zero.
- MovieLens requires manual binary thresholding (rating > 3) to fit the CTR task, which differs from standard rating prediction benchmarks and can alter class distribution.
- Weibo dataset sizes are approximate ('about 1 billion'), and exact temporal split boundaries may vary across implementations, affecting reproducibility.

## Evidence (verbatim from paper)

> AUC is one of the most popular evaluation metrics for CTR prediction which measures the goodness of order by ranking all the ads with predicted CTR, including intra-user and inter-user orders. We adopt RelaImpr introduced in Yan et al. (2014) to measure relative improvement over models It is defined as follows: RelaImpr=((AUC(measured model)-0.5)/(AUC(base model)-0.5)-1)×100%.

## Citation

```bibtex
@misc{niu2018structured,
  title={Structured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction},
  author={Niu et al. (2018)},
  year={2018},
  note={arXiv:1812.01353}
}
```

- arXiv: 1812.01353