Ctr Prediction Dhan Eval

---
name: ctr-prediction-dhan-eval
description: Evaluates a model's ability to predict the next item a user will click or review based on their historical interaction sequence. It probes hierarchical user interest modeling across multiple product dimensions and abstraction levels in recommendation systems. Use when the user wants to benchmark on Amazon Review (Six-Category, Kindle Shop, Electronics), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.12981
  bibtex_key: xu2020dhan
  confidence: high
---

# ctr-prediction-dhan-eval

> Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction — Weinan Xu et al. (2020) (arXiv:2005.12981, 2020)

## What this evaluates

Evaluates a model's ability to predict the next item a user will click or review based on their historical interaction sequence. It probes hierarchical user interest modeling across multiple product dimensions and abstraction levels in recommendation systems.

## Datasets

- **Amazon Review (Six-Category, Kindle Shop, Electronics)** — total ?; splits: train (-1), test (-1); repo https://github.com/stellaxu/DHAN

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (reviewed item) is ranked higher than a randomly chosen negative instance (unreviewed item).
- `RelaImpr` — range: percent
  - Relative Improvement over a baseline model: ((AUC(model) - 0.5) / (AUC(base) - 0.5) - 1) * 100%. Used to quantify performance gain relative to a reference model.

## Input / output format

**Input**: Sequence of the first k historically reviewed items (positive samples) and a candidate item (positive or negative sample) for each user.

**Output**: A predicted click-through rate (CTR) probability or relevance score for the candidate item.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def compute_rela_impr(auc_model, auc_base):
    return ((auc_model - 0.5) / (auc_base - 0.5) - 1) * 100

# y_true: 1 for reviewed, 0 for unreviewed
# y_pred: model's predicted probability
auc = compute_auc(y_true, y_pred)
rela_impr = compute_rela_impr(auc, auc_baseline)
```

## Common pitfalls

- AUC is computed on sampled negative items, so results depend heavily on the negative sampling strategy (items not reviewed by the user).
- RelaImpr measures relative gain over a specific baseline (DIN), not absolute performance; comparing across papers requires matching baselines.
- The task uses a sequential prediction setup (predict k+1 from k), so evaluation must respect temporal order to avoid data leakage.

## Evidence (verbatim from paper)

> Our task is to predict $(\mathrm{k} + 1)$ -th reviewed item by using the first $k$ reviewed items. For each user, we construct positive sample by taking user's historical reviews, and generate a negative sample from items not reviewed by this user. RelaImpr[3] metric is used to measure the relative improvement over models, which is defined as: $$ \text{RelaImpr} = \left(\frac{\text{AUC(measured model)} - 0.5}{\text{AUC(base model)} - 0.5} -1\right)\times 100\%. $$

## Citation

```bibtex
@misc{xu2020dhan,
  title={Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction},
  author={Weinan Xu et al. (2020)},
  year={2020},
  note={arXiv:2005.12981}
}
```

- arXiv: 2005.12981