Elliptic Fraud Eval

---
name: elliptic-fraud-eval
description: Evaluates the utility, robustness, and interpretability of graph-derived signals for tabular machine learning on a binary node classification task. It compares graph-augmented models against tabular baselines using statistical hypothesis testing and graph perturbation analysis to ensure reproducibility. Use when the user wants to benchmark on Elliptic, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13998
  bibtex_key: heidrich2026grapheval
  confidence: high
---

# elliptic-fraud-eval

> A Systematic Evaluation Protocol of Graph-Derived Signals for Tabular Machine Learning — Heidrich et al. (2026) (arXiv:2603.13998, 2026)

## What this evaluates

Evaluates the utility, robustness, and interpretability of graph-derived signals for tabular machine learning on a binary node classification task. It compares graph-augmented models against tabular baselines using statistical hypothesis testing and graph perturbation analysis to ensure reproducibility.

## Datasets

- **Elliptic** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graph-eval/graph-eval-protocol

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Aggregated across random seeds using trimmed aggregation to reduce outlier influence.
- `McNemar's test` — range: [0, 1]
  - Statistical hypothesis test applied to paired predictions on identical test splits to assess significance of performance differences between baseline and graph-augmented models. Results with p ≤ 0.05 are considered statistically significant.

## Input / output format

**Input**: Transaction-level features concatenated with graph-derived signals (or original features only for baseline). Binary node classification task.

**Output**: Binary prediction (licit or illicit) per node.

## Scoring recipe

```python
def compute_f1(y_true, y_pred):
    tp = sum((y_true == 1) & (y_pred == 1))
    fp = sum((y_true == 0) & (y_pred == 1))
    fn = sum((y_true == 1) & (y_pred == 0))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def mcnemar_pval(y_true, y_pred_a, y_pred_b):
    b = sum((y_true != y_pred_a) & (y_true == y_pred_b))
    c = sum((y_true == y_pred_a) & (y_true != y_pred_b))
    if b + c == 0: return 1.0
    chi2 = (abs(b - c) - 1)**2 / (b + c)
    return 1 - chi2_cdf(chi2, df=1)
```

## Common pitfalls

- Using an inductive split instead of the specified transductive setting (full graph available during training), which breaks the paired comparison assumption and alters structural information flow.
- Averaging results across seeds without using trimmed aggregation, which masks outlier runs and inflates perceived robustness.
- Reporting raw performance differences without applying McNemar's test on paired predictions, leading to false claims of statistical significance.

## Evidence (verbatim from paper)

> As the primary comparison metric, we report the F1-score, which balances precision and recall and enables direct comparison across classifiers and graph signal configurations. To assess whether observed performance differences between baseline and graph-augmented models are statistically significant, we apply McNemar’s test to paired predictions on identical test splits, considering results with p≤ 0.05 as statistically significant.

## Citation

```bibtex
@misc{heidrich2026grapheval,
  title={A Systematic Evaluation Protocol of Graph-Derived Signals for Tabular Machine Learning},
  author={Heidrich et al. (2026)},
  year={2026},
  note={arXiv:2603.13998}
}
```

- arXiv: 2603.13998