Bxaic Eval

---
name: bxaic-eval
description: Evaluates the faithfulness and localization accuracy of explainable AI (XAI) methods for Graph Neural Networks on molecular graphs. It measures how well explainers identify ground-truth chemical motifs (nodes/edges) versus correctly identifying when the entire graph is important, using threshold-free metrics. Use when the user wants to benchmark on B-XAIC, or asks about evaluating this task. Reports NE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22252
  bibtex_key: proszewska2025bxaic
  confidence: high
---

# bxaic-eval

> B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data — Proszewska et al. (2025) (arXiv:2505.22252, 2025)

## What this evaluates

Evaluates the faithfulness and localization accuracy of explainable AI (XAI) methods for Graph Neural Networks on molecular graphs. It measures how well explainers identify ground-truth chemical motifs (nodes/edges) versus correctly identifying when the entire graph is important, using threshold-free metrics.

## Datasets

- **B-XAIC** — total 50000; splits: test (-1); HF `mproszewska/B-XAIC`; repo https://github.com/mproszewska/B-XAIC

## Metrics

- `NE` **(primary)** — range: [0, 1]
  - Node Explanation score measuring the precision/localization of highlighted nodes against ground-truth chemical motifs. Computed as the overlap between explainer-attributed nodes and ground-truth substructures, averaged across all 7 chemical tasks.
- `SE` — range: [0, 1]
  - Subgraph Explanation score measuring the recall/completeness of the highlighted subgraph against ground-truth motifs. Computed as the coverage of ground-truth substructures by the explainer's output, averaged across tasks.

## Input / output format

**Input**: Molecular graphs represented as node features, edge features, and adjacency matrices for GNN processing.

**Output**: Soft masks or continuous importance scores assigned to each node and/or edge by the explainer method.

## Scoring recipe

```python
def compute_ne_se(explainer_scores, ground_truth_mask, top_k=10):
    predicted_mask = select_top_k(explainer_scores, top_k)
    tp = sum(predicted_mask & ground_truth_mask)
    ne = tp / sum(predicted_mask) if sum(predicted_mask) > 0 else 0
    se = tp / sum(ground_truth_mask) if sum(ground_truth_mask) > 0 else 0
    return ne, se
# Average NE and SE across all 7 chemical tasks and all test instances
```

## Common pitfalls

- Using a fixed threshold to binarize explainer scores across different methods, as attribution scales vary widely (e.g., near-zero vs. ~4.5).
- Assuming high predictive F1 scores guarantee faithful explanations; GIN achieves >98% F1 but explainers still fail to highlight simple patterns.
- Evaluating node and edge explanations interchangeably, as some explainers only support one modality.

## Evidence (verbatim from paper)

> Gradient-based methods are, on average, better at localizing important patterns than other methods. However, they tend to highlight molecular fragments even when the pattern is absent, resulting in low NE scores.

## Citation

```bibtex
@misc{proszewska2025bxaic,
  title={B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data},
  author={Proszewska et al. (2025)},
  year={2025},
  note={arXiv:2505.22252}
}
```

- arXiv: 2505.22252