Dcmt Cvr Eval

---
name: dcmt-cvr-eval
description: Evaluates the ability of multi-task learning models to predict post-click conversion rate (CVR) and click-through conversion rate (CTCVR) while mitigating selection bias in recommendation and search systems. It probes whether causal debiasing mechanisms improve ranking quality on both clicked and unclicked items across diverse e-commerce and industrial datasets. Use when the user wants to benchmark on Ali-CCP, Ali-Express (AE-ES), Ali-Express (AE-FR), Ali-Express (AE-NL), Ali-Express (AE-US), Alipay Search, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.06141
  bibtex_key: zhu2023dcmt
  confidence: high
---

# dcmt-cvr-eval

> DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation — Feng Zhu et al. (2023) (arXiv:2302.06141, 2023)

## What this evaluates

Evaluates the ability of multi-task learning models to predict post-click conversion rate (CVR) and click-through conversion rate (CTCVR) while mitigating selection bias in recommendation and search systems. It probes whether causal debiasing mechanisms improve ranking quality on both clicked and unclicked items across diverse e-commerce and industrial datasets.

## Datasets

- **Ali-CCP** — total ?; splits: (unstated)
- **Ali-Express (AE-ES)** — total ?; splits: (unstated)
- **Ali-Express (AE-FR)** — total ?; splits: (unstated)
- **Ali-Express (AE-NL)** — total ?; splits: (unstated)
- **Ali-Express (AE-US)** — total ?; splits: (unstated)
- **Alipay Search** — total ?; splits: train (7), val (1), test (2)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Computed as the integral of the true positive rate versus false positive rate across all classification thresholds. Ranges from 0 to 1, where 1 indicates perfect ranking.
- `PV-CTR` — range: percent
  - Click-through rate per page view, calculated as total clicks divided by total page views in the online experiment.
- `PV-CVR` — range: percent
  - Conversion rate per page view, calculated as total conversions divided by total page views.
- `Top-5 PV-CVR` — range: percent
  - Conversion rate per page view restricted to the top-5 displayed services on a screen.

## Input / output format

**Input**: User features, item features, combination features, context features, and exposure features representing the clicked or non-clicked items in the recommendation list.

**Output**: Continuous probability scores for click (CTR) and conversion (CVR) for each candidate item.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auc(y_true, y_pred):
    # y_true: binary labels (0/1) for click or conversion
    # y_pred: predicted probabilities
    if len(np.unique(y_true)) < 2:
        return 0.0
    return roc_auc_score(y_true, y_pred)

# Offline evaluation
auc_cvr = compute_auc(gold_cvr_labels, pred_cvr_scores)
auc_ctcvr = compute_auc(gold_ctcvr_labels, pred_ctcvr_scores)
# Average over 5 random seeds
```

## Common pitfalls

- CVR labels are only observable for clicked items (factual space), so naive offline evaluation on clicked data severely overestimates performance due to selection bias.
- Online business metrics (PV-CTR, PV-CVR) measure system-level impact, while offline AUC measures ranking quality; improvements in AUC do not always translate to online metric gains.
- Datasets are proprietary industrial logs with encrypted features, preventing exact feature reproduction or public dataset replication without platform access.

## Evidence (verbatim from paper)

> The area under the ROC curve (AUC) is adopted as performance metrics in the experiments. All experiments are repeated 5 times and averaged results are reported.

## Citation

```bibtex
@misc{zhu2023dcmt,
  title={DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation},
  author={Feng Zhu et al. (2023)},
  year={2023},
  note={arXiv:2302.06141}
}
```

- arXiv: 2302.06141