Gnn Explainer Eval

---
name: gnn-explainer-eval
description: Evaluates the quality and interpretability of explanations generated by various Graph Neural Network (GNN) explainers across different architectures and graph datasets. It probes how well explanations align with human expectations (plausibility) and model decision logic (fidelity). Use when the user wants to benchmark on Grid, Grid-House, Stars, House-Color, or asks about evaluating this task. Reports F1-Fidelity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.15304
  bibtex_key: longa2022explaining
  confidence: medium
---

# gnn-explainer-eval

> Explaining the Explainers in Graph Neural Networks: a Comparative Study — Longa et al. (2022) (arXiv:2210.15304, 2022)

## What this evaluates

Evaluates the quality and interpretability of explanations generated by various Graph Neural Network (GNN) explainers across different architectures and graph datasets. It probes how well explanations align with human expectations (plausibility) and model decision logic (fidelity).

## Datasets

- **Grid** — total ?; splits: test (-1)
- **Grid-House** — total ?; splits: test (-1)
- **Stars** — total ?; splits: test (-1)
- **House-Color** — total ?; splits: test (-1)

## Metrics

- `F1-Fidelity` **(primary)** — range: [0, 1]
  - Measures the overlap between the explanation mask and the model's actual decision logic, typically computed as an F1 score between the predicted important subgraph and the model's attention or gradient map.
- `Plausibility` — range: [0, 1]
  - Measures how close the explanation is to human-expected ground truth or intuitive concepts, often evaluated via human studies or predefined structural benchmarks.

## Input / output format

**Input**: Graph structures (nodes, edges, features) for node or graph classification tasks.

**Output**: Explanation masks indicating node or edge importances (continuous weights or binary masks).

## Scoring recipe

```python
def evaluate(explainer, model, dataset):
    mask = explainer.explain(model, dataset)
    fidelity = compute_f1_fidelity(mask, model_predictions)
    plausibility = compute_plausibility(mask, human_ground_truth)
    aggregated = aggregate_scores(fidelity, plausibility)
    return aggregated
```

## Common pitfalls

- Overreliance on consistency metrics without considering structural fidelity.
- Node-based explainers may outperform edge-based ones on node classification but underperform on graph classification.
- Aggregating scores across diverse datasets can mask dataset-specific explainability challenges.

## Evidence (verbatim from paper)

> With this question we would like to understand which is the architecture that achieves the best score, either in terms of f1-fidelity or plausibility.

## Citation

```bibtex
@misc{longa2022explaining,
  title={Explaining the Explainers in Graph Neural Networks: a Comparative Study},
  author={Longa et al. (2022)},
  year={2022},
  note={arXiv:2210.15304}
}
```

- arXiv: 2210.15304