Coat Ranking Eval

---
name: coat-ranking-eval
description: Evaluates the ability of debiasing frameworks to transform biased (MNAR) recommendation data into unbiased (MAR) representations, measuring how well debiased rankings align with ground-truth user preferences. It probes whether reweighting or perturbation mechanisms successfully mitigate selection and staleness biases without degrading predictive performance. Use when the user wants to benchmark on Coat, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.03997
  bibtex_key: khatami2025robust
  confidence: high
---

# coat-ranking-eval

> Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems — Khatami et al. (2025) (arXiv:2504.03997, 2025)

## What this evaluates

Evaluates the ability of debiasing frameworks to transform biased (MNAR) recommendation data into unbiased (MAR) representations, measuring how well debiased rankings align with ground-truth user preferences. It probes whether reweighting or perturbation mechanisms successfully mitigate selection and staleness biases without degrading predictive performance.

## Datasets

- **Coat** — total 11600; splits: train (6960), test (4640)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance (rating ≥4) is ranked higher than a randomly chosen negative instance (rating <4).
- `Precision` — range: [0, 1]
  - Fraction of predicted positive clicks that are actually positive: TP / (TP + FP).
- `Recall` — range: [0, 1]
  - Fraction of actual positive clicks that are correctly predicted: TP / (TP + FN).
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
- `Wasserstein Distance` — range: other
  - Earth Mover's Distance between the predicted click distribution P(C|X^r) and the true distribution P(C|X^r, X^{nr}). Quantifies the distributional shift after debiasing.

## Input / output format

**Input**: User and item feature vectors (X^r), exposure/bias features (X^{nr}), and propensity scores. Explicit 1-5 ratings are binarized as positive (≥4) or negative (<4) for evaluation.

**Output**: Predicted click probability or ranking score for each user-item pair.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    # predictions and gold_labels are binary (0/1) or probabilities
    tp = sum(p and g for p, g in zip(predictions, gold_labels))
    fp = sum(p and not g for p, g in zip(predictions, gold_labels))
    fn = sum(not p and g for p, g in zip(predictions, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(gold_labels, predictions)
    return {'AUC': auc, 'Precision': precision, 'Recall': recall, 'F1': f1}
```

## Common pitfalls

- Confusing biased (MNAR) training/evaluation data with unbiased (MAR) golden data, leading to overoptimistic metric scores that do not reflect true user preference.
- Using sparse down-funnel signals like 'saves' for model training or direct evaluation instead of debiasing the more prevalent click data, which causes instability.
- Failing to binarize explicit 1-5 ratings into ≥4 vs <4 before computing Precision/Recall/F1, violating the paper's evaluation protocol.

## Evidence (verbatim from paper)

> Our method (E5) shows the lowest drift from E1 in AUC and F1-score, which balances recall and precision, compared to all baselines.

## Citation

```bibtex
@misc{khatami2025robust,
  title={Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems},
  author={Khatami et al. (2025)},
  year={2025},
  note={arXiv:2504.03997}
}
```

- arXiv: 2504.03997