Dhen Ctr Eval

---
name: dhen-ctr-eval
description: Evaluates the effectiveness of a deep hierarchical ensemble network for large-scale click-through rate (CTR) prediction. It probes the model's ability to capture complex, non-overlapping feature interactions across multiple layers and scale efficiently on industrial-scale data. Use when the user wants to benchmark on Industrial in-house dataset, or asks about evaluating this task. Reports Normalized Entropy (NE) loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.11014
  bibtex_key: zhang2022dhen
  confidence: high
---

# dhen-ctr-eval

> DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction — Buyun Zhang et al. (2022) (arXiv:2203.11014, 2022)

## What this evaluates

Evaluates the effectiveness of a deep hierarchical ensemble network for large-scale click-through rate (CTR) prediction. It probes the model's ability to capture complex, non-overlapping feature interactions across multiple layers and scale efficiently on industrial-scale data.

## Datasets

- **Industrial in-house dataset** — total ?; splits: train (-1)

## Metrics

- `Normalized Entropy (NE) loss` **(primary)** — range: percent
  - Normalized Entropy loss is used to evaluate CTR prediction accuracy. The paper reports the relative difference in NE loss compared to a baseline at specific training steps (e.g., @10B examples). Lower NE indicates better performance.

## Input / output format

**Input**: Hundreds of sparse (categorical) features and thousands of dense (numerical) features per user/ad impression instance.

**Output**: CTR prediction probability.

## Scoring recipe

```python
baseline_ne = compute_normalized_entropy_loss(baseline_preds, labels)
model_ne = compute_normalized_entropy_loss(model_preds, labels)
# Evaluated at fixed training steps (e.g., 10B, 20B examples)
ne_diff_pct = ((model_ne - baseline_ne) / baseline_ne) * 100
return ne_diff_pct  # Negative values indicate improvement over baseline
```

## Common pitfalls

- Normalized Entropy is a loss metric where lower values are better, but the paper reports 'NE diff' (relative improvement), which can be misinterpreted as a higher-is-better accuracy score.
- The dataset is an in-house industrial dataset with no public splits or size information, making direct replication impossible without internal access.
- Performance is measured at fixed training steps (number of examples processed) rather than fixed epochs or wall-clock time, which affects convergence comparisons.

## Evidence (verbatim from paper)

> To facilitate comparison, we use DCN as baseline and use relative Normalized Entropy loss difference at different training steps (training examples) to evaluate the model performance.

## Citation

```bibtex
@misc{zhang2022dhen,
  title={DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction},
  author={Buyun Zhang et al. (2022)},
  year={2022},
  note={arXiv:2203.11014}
}
```

- arXiv: 2203.11014