Semignn Alipay Eval

---
name: semignn-alipay-eval
description: Evaluates a semi-supervised graph neural network's ability to predict user loan defaults and classify user occupations using multiview graph data (social ties, app usage, nicks, addresses) on a large-scale financial platform dataset. Use when the user wants to benchmark on Alipay, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.01171
  bibtex_key: wang2020semignn
  confidence: high
---

# semignn-alipay-eval

> A Semi-supervised Graph Attentive Network for Financial Fraud Detection — Wang et al. (2020) (arXiv:2003.01171, 2020)

## What this evaluates

Evaluates a semi-supervised graph neural network's ability to predict user loan defaults and classify user occupations using multiview graph data (social ties, app usage, nicks, addresses) on a large-scale financial platform dataset.

## Datasets

- **Alipay** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between default and non-default users across all classification thresholds.
- `KS` — range: [0, 1]
  - Kolmogorov-Smirnov statistic. Measures risk differentiation by finding the maximum difference between the cumulative distribution functions of positive (default) and negative (non-default) classes.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall. Used for multi-class occupation prediction.
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive occupations to all predicted positives.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive occupations to all actual positives.
- `Top-1% Precision` — range: [0, 1]
  - Precision calculated only on the top 1% of users ranked by predicted risk/occupation probability, focusing on high-confidence predictions for financial risk control.

## Input / output format

**Input**: Multiview graph data per user: user-relation graph (social ties), user-app graph (app login frequency), user-nick graph (nick words), user-address graph (address words). Node features are derived from pretrained embeddings of these views. Labels are provided for a subset of users (default/non-default or occupation).

**Output**: Binary prediction for user default (default vs non-default) or multi-class prediction for user occupation.

## Scoring recipe

```python
def compute_auc(y_true, y_score):
    return sklearn.metrics.roc_auc_score(y_true, y_score)

def compute_ks(y_true, y_score):
    order = np.argsort(y_score)[::-1]
    y_true_sorted = y_true[order]
    pos_cum = np.cumsum(y_true_sorted)
    neg_cum = np.cumsum(1 - y_true_sorted)
    total_pos = pos_cum[-1]
    total_neg = neg_cum[-1]
    return np.max(np.abs(pos_cum/total_pos - neg_cum/total_neg))
```

## Common pitfalls

- The dataset is highly imbalanced (only 5% labeled as default), making accuracy misleading; AUC and KS are preferred for risk differentiation.
- Evaluation splits (50/20/30) apply only to the 4M labeled users, while >100M unlabeled users are used for semi-supervised training but excluded from metric calculation.
- Top-1% Precision is a domain-specific metric for financial risk control, not a standard ML metric, and may be misinterpreted as standard precision.

## Evidence (verbatim from paper)

> Commonly, we use AUC as the evaluation metric. Specifically, financial scenario also concerns about the KS, which is a metric to measure the risk differentiation of the model.

## Citation

```bibtex
@misc{wang2020semignn,
  title={A Semi-supervised Graph Attentive Network for Financial Fraud Detection},
  author={Wang et al. (2020)},
  year={2020},
  note={arXiv:2003.01171}
}
```

- arXiv: 2003.01171