Yelpchi Fraud Detection Eval

---
name: yelpchi-fraud-detection-eval
description: This evaluation probes a graph neural network's ability to detect fraudulent or spam reviews within a multi-relational graph structure. It measures classification robustness against structural and semantic inconsistencies by training on varying fractions of labeled data and testing on the remainder. Use when the user wants to benchmark on YelpChi, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.00625
  bibtex_key: liu2020alleviating
  confidence: high
---

# yelpchi-fraud-detection-eval

> Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detection — Zhiwei Liu et al. (arXiv:2005.00625, 2020)

## What this evaluates

This evaluation probes a graph neural network's ability to detect fraudulent or spam reviews within a multi-relational graph structure. It measures classification robustness against structural and semantic inconsistencies by training on varying fractions of labeled data and testing on the remainder.

## Datasets

- **YelpChi** — total 45954; splits: train (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Measures overall classification performance on the imbalanced spam/legitimate review task.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to rank spam reviews higher than legitimate ones across all classification thresholds.

## Input / output format

**Input**: Graph-structured data where nodes represent reviews, each initialized with a 100-dimensional Word2Vec feature vector. Edges encode three relations: same user (R-U-R), same product & rating (R-S-R), and same product & month (R-T-R). Each node has a binary ground-truth label (spam or legitimate).

**Output**: Binary classification label (spam/legitimate) or a continuous confidence score for each review node.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_proba):
    y_pred = (y_pred_proba >= 0.5).astype(int)
    tp = ((y_pred == 1) & (y_true == 1)).sum()
    fp = ((y_pred == 1) & (y_true == 0)).sum()
    fn = ((y_pred == 0) & (y_true == 1)).sum()
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(y_true, y_pred_proba)
    return {'F1': f1, 'AUC': auc}
```

## Common pitfalls

- The evaluation uses variable training data percentages (40%, 60%, 80%) rather than a fixed train/val/test split, requiring strict random seed control for reproducibility.
- The graph construction explicitly filters products with >800 reviews and builds three specific multi-relational edges, deviating from standard single-relation node classification benchmarks.
- AUC is computed on prediction probabilities while F1 uses a fixed 0.5 threshold; reporting them interchangeably without specifying the threshold causes metric misalignment.

## Evidence (verbatim from paper)

> We use F1-score to measure the overall classification performance and AUC to measure the performance of identifying spam reviews.

## Citation

```bibtex
@misc{liu2020alleviating,
  title={Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detection},
  author={Zhiwei Liu et al.},
  year={2020},
  note={arXiv:2005.00625}
}
```

- arXiv: 2005.00625