Sec Gfd Eval

---
name: sec-gfd-eval
description: Evaluates graph neural networks for fraud detection on real-world transaction and review graphs. It specifically probes a model's robustness to severe class imbalance and heterophily, where connected nodes often belong to different classes. Use when the user wants to benchmark on Amazon, YelpChi, T-Finance, T-Social, or asks about evaluating this task. Reports F1-macro, AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.06441
  bibtex_key: xu2023revisiting
  confidence: high
---

# sec-gfd-eval

> Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum — Fan Xu et al. (2023) (arXiv:2312.06441, 2023)

## What this evaluates

Evaluates graph neural networks for fraud detection on real-world transaction and review graphs. It specifically probes a model's robustness to severe class imbalance and heterophily, where connected nodes often belong to different classes.

## Datasets

- **Amazon** — total 11944; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD
- **YelpChi** — total 45954; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD
- **T-Finance** — total 39357; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD
- **T-Social** — total 5781065; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD

## Metrics

- `F1-macro` **(primary)** — range: [0, 100] percent
  - Weighted average of F1 scores across all classes (fraud and normal). Reported as a percentage.
- `AUC` **(primary)** — range: [0, 100] percent
  - Area under the Receiver Operating Characteristic (ROC) curve, measuring the trade-off between true positive and false positive rates. Reported as a percentage.

## Input / output format

**Input**: Graph-structured data containing node features, adjacency matrix, and binary node labels (normal vs. fraudulent).

**Output**: Binary classification label or anomaly score for each node.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    f1_macro = f1_score(y_true, y_pred, average='macro') * 100
    auc = roc_auc_score(y_true, y_pred) * 100
    return f1_macro, auc
```

## Common pitfalls

- The train/val/test split ratio is fixed at 0.4/0.2/0.4, which differs from the standard 0.6/0.2/0.2 used in many node classification benchmarks.
- Datasets are highly imbalanced (3–14% anomalies), making accuracy an unreliable metric; F1-macro and AUC must be used.
- Heterophily is prevalent, meaning standard low-pass GNNs (e.g., GCN) are expected to underperform compared to band-pass or high-pass variants.

## Evidence (verbatim from paper)

> As graph anomaly detection poses a class-imbalanced classification problem, this paper utilizes two widely adopted metrics: F1-macro and AUC. F1-macro considers the weighted average of F1 scores across multiple classes, and AUC is the area under the ROC Curve.

## Citation

```bibtex
@misc{xu2023revisiting,
  title={Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum},
  author={Fan Xu et al. (2023)},
  year={2023},
  note={arXiv:2312.06441}
}
```

- arXiv: 2312.06441