Esmm Cvr Eval

---
name: esmm-cvr-eval
description: Evaluates multi-task learning architectures for post-click conversion rate (CVR) prediction in online advertising, specifically testing how parameter sharing and entire-space training mitigate data sparsity and bias. Use when the user wants to benchmark on Unspecified industry advertising dataset, or asks about evaluating this task. Reports Performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.13475
  bibtex_key: obrien2021esmm
  confidence: medium
---

# esmm-cvr-eval

> An Analysis Of Entire Space Multi-Task Models For Post-Click Conversion Prediction — Conor O'Brien et al. (2021) (arXiv:2108.13475, 2021)

## What this evaluates

Evaluates multi-task learning architectures for post-click conversion rate (CVR) prediction in online advertising, specifically testing how parameter sharing and entire-space training mitigate data sparsity and bias.

## Datasets

- **Unspecified industry advertising dataset** — total ?; splits: test (-1)

## Metrics

- `Performance` **(primary)** — range: relative (baseline IP = 1.0)
  - Normalized by the mean score of the IP baseline model (IP = 1.0). Calculated across at least 10 independent runs and reported as mean ± SEM.

## Input / output format

**Input**: Not specified in the provided text.

**Output**: Not specified in the provided text.

## Scoring recipe

```python
scores = []
for run in range(10):
    model_preds = predict(model, test_data)
    baseline_preds = predict(IP_model, test_data)
    model_score = compute_metric(model_preds, gold)
    baseline_score = compute_metric(baseline_preds, gold)
    scores.append(model_score / baseline_score)
mean_val = mean(scores)
sem = std(scores) / sqrt(len(scores))
return mean_val, sem
```

## Common pitfalls

- Hyperparameter tuning significantly impacts ESMM performance more than other models, requiring careful search.
- Data bias requires explicit weighting (e.g., ESMM-NS) or parameter sharing to avoid learning noise from sparse positive install labels.
- Statistical significance must be verified via 2-sided t-tests across ≥10 independent runs due to high variance.

## Evidence (verbatim from paper)

> Figure 3 gives a summary of the key results from our experiments. They provide clear evidence that a meaningful decomposition of the prediction task has clear benefits, shown by ESP performing 2% worse than IP. ... The standard error for each model's performance difference (against IP mean) is calculated across at least 10 runs. The results show that there are multiple mechanisms for inducing positive transfer, but that hard parameter sharing alone (IPSP) may be optimal. Better than column indicates models that this model outperforms p < 0.01, 2-sided t-test.

## Citation

```bibtex
@misc{obrien2021esmm,
  title={An Analysis Of Entire Space Multi-Task Models For Post-Click Conversion Prediction},
  author={Conor O'Brien et al. (2021)},
  year={2021},
  note={arXiv:2108.13475}
}
```

- arXiv: 2108.13475