Xfraud Fraud Detection Eval

---
name: xfraud-fraud-detection-eval
description: Evaluates the ability of graph neural networks to detect fraudulent transactions in large-scale, highly imbalanced e-commerce transaction graphs. It probes model performance under extreme class imbalance and measures the trade-off between detection accuracy, inference speed, and scalability across different graph sizes and distributed settings. Use when the user wants to benchmark on eBay-xlarge, eBay-large, eBay-small, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.12193
  bibtex_key: rao2020xfraud
  confidence: high
---

# xfraud-fraud-detection-eval

> xFraud: Explainable Fraud Transaction Detection — Rao et al. (2020) (arXiv:2011.12193, 2020)

## What this evaluates

Evaluates the ability of graph neural networks to detect fraudulent transactions in large-scale, highly imbalanced e-commerce transaction graphs. It probes model performance under extreme class imbalance and measures the trade-off between detection accuracy, inference speed, and scalability across different graph sizes and distributed settings.

## Datasets

- **eBay-xlarge** — total ?; splits: test (-1); repo https://github.com/eBay/xFraud
- **eBay-large** — total ?; splits: test (-1); repo https://github.com/eBay/xFraud
- **eBay-small** — total ?; splits: test (-1); repo https://github.com/eBay/xFraud

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `Inference time` — range: seconds
  - Wall-clock time required to process a batch of 640 nodes during the testing phase.
- `Training time` — range: seconds
  - Wall-clock time required to complete one full training epoch.

## Input / output format

**Input**: Heterogeneous transaction graphs containing nodes (e.g., transactions, addresses, payment tokens) and edges, with node features ranging from 114 to 480 dimensions depending on the dataset subset.

**Output**: Binary classification prediction per node/transaction (fraudulent vs. benign), along with continuous probability scores used to compute ROC and precision-recall curves.

## Scoring recipe

```python
def compute_auc(y_true, y_scores):
    # y_true: binary labels (1=fraud, 0=benign)
    # y_scores: predicted probabilities for fraud
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)
```

## Common pitfalls

- The evaluation uses pre-filtered and down-sampled benign transactions, which may not reflect the raw production data distribution.
- Distributed training on 16 machines yields lower AUC than 8 machines due to restricted neighbor fields, highlighting a scalability-accuracy trade-off that readers might overlook.
- Extreme class imbalance (~4% fraud) makes accuracy misleading; AUC and precision-recall curves are necessary for proper assessment.

## Evidence (verbatim from paper)

> From Table 3, our detector+, achieves the best AUC (averaged across seeds) using 8 machines w.r.t. GEM and GAT. In terms of training efficiency, xFraud detector+ takes only slightly longer time per epoch compared to GEM in an 8-machine setting.

## Citation

```bibtex
@misc{rao2020xfraud,
  title={xFraud: Explainable Fraud Transaction Detection},
  author={Rao et al. (2020)},
  year={2020},
  note={arXiv:2011.12193}
}
```

- arXiv: 2011.12193