Grl Perturbation Sensitivity Eval

---
name: grl-perturbation-sensitivity-eval
description: Evaluates graph neural network robustness and feature/structure reliance by measuring performance degradation under 13 structured perturbations to node features and graph topology. It classifies datasets based on their sensitivity profiles to structural vs. feature information. Use when the user wants to benchmark on GRL Benchmark Collection (49 datasets), or asks about evaluating this task. Reports sensitivity_profile.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.07729
  bibtex_key: liu2022taxonomy
  confidence: high
---

# grl-perturbation-sensitivity-eval

> Taxonomy of Benchmarks in Graph Representation Learning — Liu et al. (2022) (arXiv:2206.07729, 2022)

## What this evaluates

Evaluates graph neural network robustness and feature/structure reliance by measuring performance degradation under 13 structured perturbations to node features and graph topology. It classifies datasets based on their sensitivity profiles to structural vs. feature information.

## Datasets

- **GRL Benchmark Collection (49 datasets)** — total 49; splits: (unstated); repo https://github.com/G-Taxonomy-Workgroup/GTaxoGym

## Metrics

- `sensitivity_profile` **(primary)** — range: [0, 1]
  - Computed as 1 - (performance_perturbed / performance_original), where performance is the task-specific metric (e.g., accuracy or AUROC) on the perturbed vs. original graph. Measures the fractional performance loss due to a specific perturbation.

## Input / output format

**Input**: Graph data consisting of adjacency matrices and node feature matrices, formatted for either node classification (transductive/inductive) or graph classification (inductive) tasks.

**Output**: Predicted node labels for each node or a single graph-level class probability for the entire graph.

## Scoring recipe

```python
def compute_sensitivity_profile(original_score, perturbed_scores):
    profile = {}
    for perturbation, perturbed_score in perturbed_scores.items():
        profile[perturbation] = 1.0 - (perturbed_score / original_score)
    return profile
```

## Common pitfalls

- Assuming structural perturbations significantly impact transductive node classification tasks, as they uniformly show high robustness to topology changes.
- Relying solely on synthetic datasets for evaluation, as they do not capture the complexity of real-world graph data and lie at the periphery of sensitivity profiles.
- Interpreting sensitivity profiles as model-specific, whereas the taxonomy shows profiles are robust across different GNN architectures (Pearson correlation > 90%).

## Evidence (verbatim from paper)

> For each dataset and perturbation combination, we show the GCN model performance relative to its performance on the unmodified dataset. Sensitivity profiles of all transductive datasets show high robustness to all graph structure perturbations.

## Citation

```bibtex
@misc{liu2022taxonomy,
  title={Taxonomy of Benchmarks in Graph Representation Learning},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2206.07729}
}
```

- arXiv: 2206.07729