Fairness Aware Gnn Eval

---
name: fairness-aware-gnn-eval
description: Evaluates the trade-off between prediction accuracy and statistical fairness for graph neural networks on node classification tasks. It probes how different in-processing and preprocessing methods, backbone architectures, and early stopping conditions affect both standard performance metrics and fairness constraints across synthetic, social, and knowledge graph datasets. Use when the user wants to benchmark on Credit, Bail, Pokec-n, Pokec-z, Pokec-n-Large, Pokec-z-Large, DBpedia, YAGO, Wikidata, or asks about evaluating this task. Reports ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.18473
  bibtex_key: sasaki2025benchmarkingfairness
  confidence: high
---

# fairness-aware-gnn-eval

> Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs — Sasaki (2025) (arXiv:2510.18473, 2025)

## What this evaluates

Evaluates the trade-off between prediction accuracy and statistical fairness for graph neural networks on node classification tasks. It probes how different in-processing and preprocessing methods, backbone architectures, and early stopping conditions affect both standard performance metrics and fairness constraints across synthetic, social, and knowledge graph datasets.

## Datasets

- **Credit** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Bail** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Pokec-n** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Pokec-z** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Pokec-n-Large** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Pokec-z-Large** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **DBpedia** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **YAGO** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase
- **Wikidata** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/yuya-s/MUSUBI-FairGraphBase

## Metrics

- `ACC` **(primary)** — range: [0, 1]
  - Standard classification accuracy: fraction of correctly predicted labels out of total nodes.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall.
- `ΔSP` — range: [0, 1]
  - Statistical Parity difference: |P(ŷ=1|s=0) - P(ŷ=1|s=1)|. Smaller is better.
- `ΔEop` — range: [0, 1]
  - Equal Opportunity difference: |P(ŷ=1|y=1,s=0) - P(ŷ=1|y=1,s=1)|. Smaller is better.

## Input / output format

**Input**: Graph adjacency structure, node feature matrix, node-level sensitive attribute vector (s ∈ {0,1}), and node-level ground truth label vector (y ∈ {0,1}) for a node classification task.

**Output**: Binary predicted label vector (ŷ ∈ {0,1}) for each node in the dataset.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, sensitive):
    acc = (y_true == y_pred).mean()
    f1 = f1_score(y_true, y_pred, average='binary')
    auc = roc_auc_score(y_true, y_pred)
    sp = abs((y_pred[sensitive == 0].mean()) - (y_pred[sensitive == 1].mean()))
    eop = abs((y_pred[(y_true == 1) & (sensitive == 0)].mean()) - 
              (y_pred[(y_true == 1) & (sensitive == 1)].mean()))
    return {'ACC': acc, 'F1': f1, 'AUC': auc, 'ΔSP': sp, 'ΔEop': eop}
```

## Common pitfalls

- Early stopping conditions drastically alter the accuracy-fairness trade-off, yet many baseline implementations use default or inconsistent stopping criteria.
- Fairness-aware methods can paradoxically degrade fairness metrics while improving accuracy, contradicting their design goals.
- Certain methods (e.g., FairGB, BIND) are incompatible with specific backbones or fail to converge within practical time/memory limits on larger graphs.

## Evidence (verbatim from paper)

> We use three prediction accuracy metrics: Accuracy (ACC), AUC, and F1. According to prior studies*([beutel2017data,] ; [TheVariationalFairAutoencoder,] )*, we use  $\Delta SP$*([dwork2012fairness,] )* and  $\Delta Eop$*([hardt2016equality,] )* to quantitatively evaluate statistical parity and equal opportunity.  $\Delta SP\=|P(\hat{y}\=1\mid s\=0)-P(\hat{y}\=1\mid s\=1)|$ and  $\Delta Eop\=|P(\hat{y}\=1\mid y\=1,s\=0)-P(\hat{y}\=1\mid y\=1,s\=1)|$, where  $y\in{0,1}$ represents the label, variable  $s\in{0,1}$ represents the sensitive attribute, and  $\hat{y}\in{0,1}$ indicates prediction output. Both  $\Delta SP$ and  $\Delta Eop$ are “smaller-is-better” metrics ( $\downarrow$), and ACC, AUC, and F1 are “larger-is-better” metrics ( $\uparrow$).

## Citation

```bibtex
@misc{sasaki2025benchmarkingfairness,
  title={Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs},
  author={Sasaki (2025)},
  year={2025},
  note={arXiv:2510.18473}
}
```

- arXiv: 2510.18473