Gnn Architecture Eval

---
name: gnn-architecture-eval
description: This evaluation probes how graph neural network (GNN) performance depends on the ratio of observed training nodes to feature dimensionality (N_obs/D). It tests whether standard benchmarks are biased toward low-dimensional regimes and evaluates the effectiveness of decoupling feature extraction from graph propagation. Use when the user wants to benchmark on Cora, Citeseer, Pubmed, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.05384
  bibtex_key: [REDACTED]
  confidence: high
---

# gnn-architecture-eval

> On the choice of graph neural network architectures — Vignac et al. (2019) (arXiv:1911.05384, 2019)

## What this evaluates

This evaluation probes how graph neural network (GNN) performance depends on the ratio of observed training nodes to feature dimensionality (N_obs/D). It tests whether standard benchmarks are biased toward low-dimensional regimes and evaluates the effectiveness of decoupling feature extraction from graph propagation.

## Datasets

- **Cora, Citeseer, Pubmed** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Accuracy is the fraction of correctly classified nodes in the test set. Results are reported as the mean accuracy over 100 random train-test splits, along with the 95% confidence interval.

## Input / output format

**Input**: Graph adjacency matrix and node feature matrix X (optionally transformed via random sketching to target dimension D').

**Output**: Predicted class label for each node in the test set.

## Scoring recipe

```python
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)
# Reported as mean(accuracies) ± 95% CI over 100 random 50% train splits
```

## Common pitfalls

- Assuming standard benchmark configurations (low N_obs/D) generalize to high-data regimes where complex GNNs are needed.
- Ignoring that feature dimensions are artificially manipulated via random sketching, which changes the effective input distribution.
- Confusing the number of propagation steps (K=2) with the number of learnable layers in the model.

## Evidence (verbatim from paper)

> We repeat experiments with different train-test splits (selected uniformly at random) for each configuration of parameters to avoid overfitting to a particular training scenario. ... Table 2: Accuracy and 95% confidence interval over 100 splits with 300 features and 50% of the nodes in the training set.

## Citation

```bibtex
@misc{vignac2019choice,
  title={On the choice of graph neural network architectures},
  author={Vignac et al. (2019)},
  year={2019},
  note={arXiv:1911.05384}
}
```

- arXiv: 1911.05384