Mac Cvr Eval

---
name: mac-cvr-eval
description: This benchmark evaluates a model's ability to predict conversion rates for ad clicks under multiple attribution mechanisms. It probes ranking capability by measuring how well predicted probabilities distinguish positive from negative samples, both globally and per user. The task treats conversion prediction as a weighted binary classification problem where continuous attribution weights act as sample importance weights. Use when the user wants to benchmark on MAC, or asks about evaluating this task. Reports GAUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.02184
  bibtex_key: wu2026mac
  confidence: high
---

# mac-cvr-eval

> MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms — Jinqi Wu et al. (2026) (arXiv:2603.02184, 2026)

## What this evaluates

This benchmark evaluates a model's ability to predict conversion rates for ad clicks under multiple attribution mechanisms. It probes ranking capability by measuring how well predicted probabilities distinguish positive from negative samples, both globally and per user. The task treats conversion prediction as a weighted binary classification problem where continuous attribution weights act as sample importance weights.

## Datasets

- **MAC** — total 79000000; splits: (unstated); HF `alimamaTech/MAC`

## Metrics

- `AUC` — range: [0, 1]
  - Area Under the ROC Curve. Computed as (1/(n+ * n-)) * sum_{i=1}^{n+} sum_{j=1}^{n-} [I(y_i^+ > y_j^-) + 0.5 * I(y_i^+ = y_j^-)], where n+ and n- are counts of positive and negative samples, and y^+ and y^- are predicted probabilities.
- `GAUC` **(primary)** — range: [0, 1]
  - Group AUC. Computed as sum_{u=1}^{U} (#click(u) * AUC_u) / sum_{u=1}^{U} #click(u), where U is the number of users, #click(u) is the number of clicks for user u, and AUC_u is the AUC computed on samples from user u. Serves as the primary evaluation criterion.

## Input / output format

**Input**: Categorical features (7 user, 10 item, 3 context), behavior sequence features (user purchase item sequence truncated to max length 20, with item/shop/category IDs and visual similarity scores), and multi-attribution continuous weights for four mechanisms (last-click, first-click, linear, DDA).

**Output**: A single continuous predicted conversion probability ŷ^{A_t} for the designated target attribution mechanism.

## Scoring recipe

```python
def compute_auc(pos_preds, neg_preds):
    ties = sum(1 for p in pos_preds for n in neg_preds if p == n)
    wins = sum(1 for p in pos_preds for n in neg_preds if p > n)
    return (wins + 0.5 * ties) / (len(pos_preds) * len(neg_preds))

def compute_gauc(user_preds, user_labels, user_click_counts):
    gauc_num = 0.0
    total_clicks = 0
    for u in range(len(user_preds)):
        pos = [p for p, l in zip(user_preds[u], user_labels[u]) if l == 1]
        neg = [p for p, l in zip(user_preds[u], user_labels[u]) if l == 0]
        auc_u = compute_auc(pos, neg)
        clicks = user_click_counts[u]
        gauc_num += clicks * auc_u
        total_clicks += clicks
    return gauc_num / total_clicks
```

## Common pitfalls

- Treating continuous attribution weights as hard binary labels without applying them as sample importance weights in the loss or evaluation.
- Computing AUC globally instead of per-user, which fails to capture the user-level ranking alignment that GAUC is designed to measure.
- Confusing auxiliary attribution mechanisms with the target mechanism; only the target mechanism's weights are used for the primary evaluation task.

## Evidence (verbatim from paper)

> For evaluating model performance, we report the widely-adopted AUC and Group AUC (GAUC) metrics of the primary task. AUC is a widely used ranking metric that evaluates a model’s ability to distinguish positive and negative samples, making it well-suited for CVR prediction. Moreover, we report the GAUC metric, which captures the model’s ability to rank ad clicks within individual user groups and serves as the primary evaluation criterion in our production system.

## Citation

```bibtex
@misc{wu2026mac,
  title={MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms},
  author={Jinqi Wu et al. (2026)},
  year={2026},
  note={arXiv:2603.02184}
}
```

- arXiv: 2603.02184