Online Ctr Eval

---
name: online-ctr-eval
description: This protocol evaluates an online feature interaction detection method integrated into click-through rate (CTR) prediction models. It measures predictive accuracy on streaming ad click data using chronological splits to simulate real-time recommendation scenarios. Use when the user wants to benchmark on Avazu, Criteo, Taobao, or asks about evaluating this task. Reports AUC, logloss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.15400
  bibtex_key: lin2021online
  confidence: high
---

# online-ctr-eval

> Online Interaction Detection for Click-Through Rate Prediction — Lin et al. (2021) (arXiv:2106.15400, 2021)

## What this evaluates

This protocol evaluates an online feature interaction detection method integrated into click-through rate (CTR) prediction models. It measures predictive accuracy on streaming ad click data using chronological splits to simulate real-time recommendation scenarios.

## Datasets

- **Avazu** — total 40428967; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)
- **Criteo** — total 45840617; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)
- **Taobao** — total 26557961; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
- `logloss` **(primary)** — range: other
  - Binary cross-entropy loss, defined as $-\frac{1}{N}\sum_{i=1}^N [y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i)]$. Lower values indicate better calibrated probability estimates.

## Input / output format

**Input**: Tabular records containing categorical and numerical features representing user context and ad properties, ordered chronologically.

**Output**: A single continuous probability value indicating the likelihood of a user clicking the displayed ad.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def compute_logloss(y_true, y_pred):
    eps = 1e-15
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
```

## Common pitfalls

- Chronological splitting is mandatory; random shuffling breaks the streaming simulation and inflates performance.
- In CTR benchmarks, improvements as small as 0.001 in AUC or logloss are considered practically significant.
- Unfreezing the base model during online fine-tuning often causes severe overfitting on validation sets that poorly represent future data.

## Evidence (verbatim from paper)

> Two common metrics for CTR prediction are adopted to evaluate the models, namely AUC (Area Under ROC) and logloss (cross-entropy). It is noticeable that a small increase of AUC or a slight decrease of logloss at 0.001-level is regarded significant for CTR prediction task, according to existing works [[3], [4], [8], [15]].

## Citation

```bibtex
@misc{lin2021online,
  title={Online Interaction Detection for Click-Through Rate Prediction},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2106.15400}
}
```

- arXiv: 2106.15400