Fairness Aware Graph Learning Eval

---
name: fairness-aware-graph-learning-eval
description: This benchmark evaluates the trade-offs between predictive utility and fairness across various graph learning algorithms. It probes how well different methods balance accuracy with demographic parity and equal opportunity constraints on real-world graph-structured data. Use when the user wants to benchmark on 7 real-world datasets, or asks about evaluating this task. Reports Δ_SP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.12112
  bibtex_key: dong2024fairnessawaregraphlearning
  confidence: medium
---

# fairness-aware-graph-learning-eval

> A Benchmark for Fairness-Aware Graph Learning — Dong et al. (2024) (arXiv:2407.12112, 2024)

## What this evaluates

This benchmark evaluates the trade-offs between predictive utility and fairness across various graph learning algorithms. It probes how well different methods balance accuracy with demographic parity and equal opportunity constraints on real-world graph-structured data.

## Datasets

- **7 real-world datasets** — total ?; splits: test (-1)

## Metrics

- `Δ_SP` **(primary)** — range: [-1, 1]
  - Statistical Parity Difference: the absolute difference in positive prediction rates between protected and unprotected groups. Lower values indicate better group fairness.
- `Δ_EO` — range: [-1, 1]
  - Equal Opportunity Difference: the absolute difference in true positive rates between protected and unprotected groups. Lower values indicate better group fairness.
- `AUC-ROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all classification thresholds.
- `running time` — range: seconds
  - Wall-clock time required to train or evaluate the model on a given dataset.

## Input / output format

**Input**: Graph-structured data containing node features, adjacency matrices, sensitive attributes, and ground-truth labels.

**Output**: Predicted node labels (or probabilities) and computed fairness/utility scores per dataset.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, sensitive_attr):
    # Group Fairness (Δ_SP)
    pred_rates = {g: mean(y_pred[sensitive_attr == g]) for g in unique(sensitive_attr)}
    delta_sp = abs(pred_rates[protected] - pred_rates[unprotected])
    # Equal Opportunity (Δ_EO)
    tpr = {g: mean((y_pred == 1) & (y_true == 1) & (sensitive_attr == g)) for g in unique(sensitive_attr)}
    delta_eo = abs(tpr[protected] - tpr[unprotected])
    # Utility
    auc_roc = roc_auc_score(y_true, y_pred)
    return delta_sp, delta_eo, auc_roc
```

## Common pitfalls

- Confusing group fairness metrics (Δ_SP, Δ_EO) with individual fairness metrics, as the benchmark evaluates both separately.
- Assuming higher utility always correlates with better fairness; the benchmark explicitly highlights method-specific trade-offs where utility sacrifices are necessary.
- Overlooking computational efficiency, as fairness-aware methods often incur significant runtime overhead compared to baseline GNNs.

## Evidence (verbatim from paper)

> According to the comprehensive empirical results, we have the consistent observation that different fairness-aware graph learning methods show different advantages in balancing utility and fairness. Specifically, we observe that GNN-based algorithms are among the highest-ranking methods (e.g., most top-ranked results come from GNN-based methods), which verifies the advantage of GNNs in achieving both utility and fairness objectives due to their exceptional fitting ability. In addition, fairness-aware shallow embedding methods achieve the best performance with respect to fairness, especially on the traditional fairness metrics  Δ_SP  and  Δ_EO .

## Citation

```bibtex
@misc{dong2024fairnessawaregraphlearning,
  title={A Benchmark for Fairness-Aware Graph Learning},
  author={Dong et al. (2024)},
  year={2024},
  note={arXiv:2407.12112}
}
```

- arXiv: 2407.12112