Trace Gmv Prediction Eval

---
name: trace-gmv-prediction-eval
description: This benchmark evaluates a model's ability to predict post-click Gross Merchandise Volume (GMV) under delayed feedback conditions. It specifically probes how well models adapt to rapidly evolving label distributions through online streaming training and whether they can effectively handle the distinct statistical properties of single-purchase versus repurchase transactions. Use when the user wants to benchmark on TRACE, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.20307
  bibtex_key: li2026trace
  confidence: high
---

# trace-gmv-prediction-eval

> Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches — Li et al. (2026) (arXiv:2601.20307, 2026)

## What this evaluates

This benchmark evaluates a model's ability to predict post-click Gross Merchandise Volume (GMV) under delayed feedback conditions. It specifically probes how well models adapt to rapidly evolving label distributions through online streaming training and whether they can effectively handle the distinct statistical properties of single-purchase versus repurchase transactions.

## Datasets

- **TRACE** — total 7160000; splits: train (-1), test (-1); repo https://github.com/alimama-tech/OnlineGMV

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (click leading to GMV) is ranked higher than a negative instance. Commonly used in ad tech to evaluate the ranking quality of predicted GMV scores.

## Input / output format

**Input**: A tuple ⟨x, t, P⟩ containing: (1) x ∈ R^d, an embedding vector of 22 categorical user/item/context features; (2) t, the click timestamp; (3) P = {(t_i^p, p_i)}_{i=1}^N, a sequence of N purchases within a 7-day attribution window, where each element contains the purchase timestamp t_i^p and transaction price p_i.

**Output**: A single scalar value ŷ representing the predicted post-click GMV.

## Scoring recipe

```python
def compute_auc(predictions, labels):
    # predictions: array of predicted GMV scores
    # labels: array of binary indicators (1 if GMV > 0, else 0)
    order = np.argsort(predictions)[::-1]
    sorted_labels = labels[order]
    tp = 0
    fp = 0
    auc = 0.0
    prev_score = None
    for i, label in enumerate(sorted_labels):
        if prev_score is not None and predictions[order[i]] != prev_score:
            auc += (tp + fp) * (i - tp)
            prev_score = predictions[order[i]]
        if label == 1:
            tp += 1
        else:
            fp += 1
    auc += (tp + fp) * (len(sorted_labels) - tp)
    num_pos = np.sum(labels == 1)
    num_neg = np.sum(labels == 0)
    return auc / (num_pos * num_neg) if (num_pos > 0 and num_neg > 0) else 0.0
```

## Common pitfalls

- Using only the final ground-truth GMV label y* for offline training ignores the delayed feedback nature of the task and fails to capture intra-day label distribution shifts.
- Directly using partially observed GMV y^(t) as the training target causes severe underestimation bias because a large portion of GMV arrives after the observation point.
- Training a single unified model on all samples ignores the significant distributional discrepancy between single-purchase and repurchase samples, hindering optimization.

## Evidence (verbatim from paper)

> As the AUC metrics listed in Table[3], the online-trained model significantly outperforms the daily-updated offline counterpart, which substantiates the significance of model freshness and the superiority of online training.

## Citation

```bibtex
@misc{li2026trace,
  title={Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.20307}
}
```

- arXiv: 2601.20307