Wide Deep Recommender Eval

---
name: wide-deep-recommender-eval
description: Evaluates a hybrid recommender model's ability to balance memorization of frequent user-item interactions with generalization to unseen combinations for ranking candidate apps. The protocol measures predictive accuracy on a static holdout set and business impact via live A/B testing on user acquisition rates. Use when the user wants to benchmark on Google Play App Store (internal), or asks about evaluating this task. Reports Online Acquisition Gain.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1606.07792
  bibtex_key: cheng2016widedeep
  confidence: high
---

# wide-deep-recommender-eval

> Wide & Deep Learning for Recommender Systems — Cheng et al. (2016) (arXiv:1606.07792, 2016)

## What this evaluates

Evaluates a hybrid recommender model's ability to balance memorization of frequent user-item interactions with generalization to unseen combinations for ranking candidate apps. The protocol measures predictive accuracy on a static holdout set and business impact via live A/B testing on user acquisition rates.

## Datasets

- **Google Play App Store (internal)** — total ?; splits: test (-1)

## Metrics

- `Offline AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve computed on a holdout set of user-app interactions. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `Online Acquisition Gain` **(primary)** — range: percent
  - Relative percentage improvement in app acquisition rate compared to the control group in a live A/B test. Calculated as (exp_rate - control_rate) / control_rate * 100%.

## Input / output format

**Input**: Sparse user and item features, including cross-product transformations and dense embeddings, representing candidate apps to be scored for ranking.

**Output**: A relevance score or probability for each candidate app, used to generate a ranked list of recommendations.

## Scoring recipe

```python
def compute_auc(y_true, y_scores):
    return roc_auc_score(y_true, y_scores)

def compute_acquisition_gain(acquisitions_control, users_control, acquisitions_exp, users_exp):
    control_rate = sum(acquisitions_control) / len(users_control)
    exp_rate = sum(acquisitions_exp) / len(users_exp)
    return (exp_rate - control_rate) / control_rate * 100
```

## Common pitfalls

- Offline AUC may not correlate strongly with online business metrics because offline data has fixed impressions and labels, whereas the online system dynamically explores and learns from new user responses.
- Online Acquisition Gain is relative to the specific control model (wide-only logistic regression); gains may not generalize to other baselines or traffic distributions.

## Evidence (verbatim from paper)

> Besides online experiments, we also show the Area Under Receiver Operator Characteristic Curve (AUC) on a holdout set offline. While Wide & Deep has a slightly higher offline AUC, the impact is more significant on online traffic. ... Wide & Deep model improved the app acquisition rate on the main landing page of the app store by +3.9% relative to the control group (statistically significant).

## Citation

```bibtex
@misc{cheng2016widedeep,
  title={Wide & Deep Learning for Recommender Systems},
  author={Cheng et al. (2016)},
  year={2016},
  note={arXiv:1606.07792}
}
```

- arXiv: 1606.07792