Dien Ctr Eval

---
name: dien-ctr-eval
description: Evaluates a model's ability to predict click-through rates (CTR) by modeling sequential user behavior and dynamically evolving latent interests relative to a target item. Use when the user wants to benchmark on Amazon Books, Amazon Electronics, Industrial (Taobao), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.03672
  bibtex_key: zhou2018dien
  confidence: high
---

# dien-ctr-eval

> Deep Interest Evolution Network for Click-Through Rate Prediction — Zhou et al. (2018) (arXiv:1809.03672, 2018)

## What this evaluates

Evaluates a model's ability to predict click-through rates (CTR) by modeling sequential user behavior and dynamically evolving latent interests relative to a target item.

## Datasets

- **Amazon Books** — total 603668; splits: train (-1), test (-1)
- **Amazon Electronics** — total 192403; splits: train (-1), test (-1)
- **Industrial (Taobao)** — total 7000000000; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen positive instance (click) is ranked higher than a randomly chosen negative instance (non-click).

## Input / output format

**Input**: User's historical behavior sequence (e.g., reviews or clicked items) up to time T-1, and a target item at time T.

**Output**: A predicted probability score indicating the likelihood of the user clicking the target item.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    pos_scores = [s for s, y in zip(y_pred, y_true) if y == 1]
    neg_scores = [s for s, y in zip(y_pred, y_true) if y == 0]
    if not pos_scores or not neg_scores:
        return 0.0
    concordant = sum(1 for p in pos_scores for n in neg_scores if p > n)
    tied = sum(1 for p in pos_scores for n in neg_scores if p == n)
    return (concordant + 0.5 * tied) / (len(pos_scores) * len(neg_scores))
```

## Common pitfalls

- The industrial dataset uses a strict temporal split (training on past 49 days, testing on the next day), so random shuffling will cause data leakage and inflate metrics.
- AUC is reported as mean ± std over 5 runs, but the paper does not specify the exact random seed or data partitioning strategy for the public datasets, making exact replication difficult.
- Online A/B testing metrics (CTR/eCPM gain) are relative to a specific baseline (BaseModel) and cannot be directly compared to offline AUC without knowing the baseline's absolute values.

## Evidence (verbatim from paper)

> We use both public and industrial datasets to verify the effect of DIEN. ... Each experiment is repeated 5 times. Table 2: Results (AUC) on public datasets ... Table 3: Results (AUC) on industrial dataset

## Citation

```bibtex
@misc{zhou2018dien,
  title={Deep Interest Evolution Network for Click-Through Rate Prediction},
  author={Zhou et al. (2018)},
  year={2018},
  note={arXiv:1809.03672}
}
```

- arXiv: 1809.03672