Pg Gnn Eval

---
name: pg-gnn-eval
description: Evaluates the expressivity and predictive performance of permutation-sensitive Graph Neural Networks (PG-GNN) on synthetic substructure counting tasks and real-world graph classification/regression benchmarks. It probes the model's ability to capture pairwise node correlations and higher-order substructures (triangles, 4-cliques) compared to standard permutation-invariant GNNs. Use when the user wants to benchmark on Erdős-Rényi random graphs, Random regular graphs, TUDataset (PROTEINS, NCI1, IMDB-B, IMDB-M, COLLAB), MNIST, ZINC, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.14368
  bibtex_key: huang2022going
  confidence: high
---

# pg-gnn-eval

> Going Deeper into Permutation-Sensitive Graph Neural Networks — Huang et al. (2022) (arXiv:2205.14368, 2022)

## What this evaluates

Evaluates the expressivity and predictive performance of permutation-sensitive Graph Neural Networks (PG-GNN) on synthetic substructure counting tasks and real-world graph classification/regression benchmarks. It probes the model's ability to capture pairwise node correlations and higher-order substructures (triangles, 4-cliques) compared to standard permutation-invariant GNNs.

## Datasets

- **Erdős-Rényi random graphs** — total ?; splits: test (-1)
- **Random regular graphs** — total ?; splits: test (-1)
- **TUDataset (PROTEINS, NCI1, IMDB-B, IMDB-M, COLLAB)** — total ?; splits: test (-1)
- **MNIST** — total ?; splits: test (-1)
- **ZINC** — total ?; splits: test (-1)

## Metrics

- `Mean Absolute Error (MAE)` — range: other
  - The average of the absolute differences between predicted and ground-truth substructure counts or regression targets. Computed as the mean of |y_pred - y_true| across all test instances.
- `Accuracy (%)` **(primary)** — range: percent
  - The percentage of correctly classified graphs out of the total test set. Computed as (number of correct predictions / total number of predictions) * 100.

## Input / output format

**Input**: Graphs represented by adjacency matrices and node feature matrices. For real-world datasets, node features are one-hot encoded categorical labels (bioinformatics) or node degrees (social networks). For synthetic tasks, graphs are Erdős-Rényi or random regular graphs with ground-truth incidence substructure counts.

**Output**: Predicted counts for incidence triangles/4-cliques (synthetic), or predicted graph class labels (TUDataset, MNIST) / regression values (ZINC).

## Scoring recipe

```python
def compute_mae(predictions, targets):
    return np.mean(np.abs(np.array(predictions) - np.array(targets)))

def compute_accuracy(predictions, targets):
    return np.mean(np.array(predictions) == np.array(targets)) * 100
```

## Common pitfalls

- Failing to replicate the exact data splits referenced in Xu et al. (for TUDataset) and Dwivedi et al. (for MNIST/ZINC), which are not fully detailed in the main text.
- Not averaging results over multiple random seeds (5 for synthetic, 4 for real-world) and reporting standard deviations as required by the protocol.
- Confusing the node-level incidence substructure counting task on synthetic graphs with the standard graph-level classification/regression tasks on real-world benchmarks.

## Evidence (verbatim from paper)

> Table 2 summarizes the results measured by Mean Absolute Error (MAE, lower is better) for incidence triangle counting. We report the average and standard deviation of testing MAEs over 5 runs with 5 different seeds. ... Table 3: Results (measured by accuracy: %) on TUDataset.

## Citation

```bibtex
@misc{huang2022going,
  title={Going Deeper into Permutation-Sensitive Graph Neural Networks},
  author={Huang et al. (2022)},
  year={2022},
  note={arXiv:2205.14368}
}
```

- arXiv: 2205.14368