Calicalcausalrank Eval

---
name: calicalcausalrank-eval
description: Evaluates multi-objective ad ranking models on click-through rate (CTR) and conversion rate (CVR) prediction tasks, focusing on ranking quality, score calibration, and counterfactual utility estimation under position and selection bias. Use when the user wants to benchmark on Criteo, Avazu, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.18786
  bibtex_key: yang2026calicausalrank
  confidence: high
---

# calicalcausalrank-eval

> CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization — Yang et al. (2026) (arXiv:2602.18786, 2026)

## What this evaluates

Evaluates multi-objective ad ranking models on click-through rate (CTR) and conversion rate (CVR) prediction tasks, focusing on ranking quality, score calibration, and counterfactual utility estimation under position and selection bias.

## Datasets

- **Criteo** — total 45000000; splits: train (-1), test (-1)
- **Avazu** — total 40000000; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring pairwise ranking quality between positive and negative samples.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring the quality of the top-k relevance ordering relative to an ideal ranking.
- `Expected Calibration Error (ECE)` — range: [0, 1]
  - Weighted average absolute difference between predicted probabilities and actual positive frequencies across K=20 calibration buckets.
- `Utility@10` — range: other
  - Counterfactual utility estimate for the top-10 ranked items, computed using variance-reduced off-policy estimators to correct for position and selection bias.

## Input / output format

**Input**: Ad impression features including 13 numerical and 26 categorical features (Criteo) or device and temporal features (Avazu), with binary labels for click (CTR) and synthetic/real conversion (CVR).

**Output**: Predicted probability scores for CTR and CVR, used to rank ads and compute ranking, calibration, and utility metrics.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, topk=10, buckets=20):
    auc = roc_auc_score(y_true, y_pred)
    ndcg = ndcg_at_k(y_true, y_pred, k=topk)
    bin_edges = np.linspace(0, 1, buckets + 1)
    ece = 0.0
    for i in range(buckets):
        mask = (y_pred >= bin_edges[i]) & (y_pred < bin_edges[i+1])
        if mask.sum() > 0:
            ece += mask.sum() * abs(y_true[mask].mean() - y_pred[mask].mean())
    ece /= len(y_true)
    utility = counterfactual_utility_estimate(y_true, y_pred, topk)
    return {'AUC': auc, 'NDCG@10': ndcg, 'ECE': ece, 'Utility@10': utility}
```

## Common pitfalls

- CVR labels are synthetically generated only for the clicked subset (~3% of samples) with a fixed 10% base probability, which may not reflect real-world conversion distributions or introduce label leakage if not handled carefully.
- Utility@10 relies on counterfactual/off-policy estimation to correct for position and selection bias, meaning scores are not directly comparable to standard online A/B test metrics without proper variance reduction.

## Evidence (verbatim from paper)

> We report: AUC for ranking quality, NDCG@10 for top-k performance, Expected Calibration Error (ECE) for calibration, and Utility@10 computed using counterfactual evaluation.

## Citation

```bibtex
@misc{yang2026calicausalrank,
  title={CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization},
  author={Yang et al. (2026)},
  year={2026},
  note={arXiv:2602.18786}
}
```

- arXiv: 2602.18786