Gnn Design Eval

---
name: gnn-design-eval
description: Evaluates an LLM-guided framework's ability to automatically propose and refine Graph Neural Network architectures for node classification across diverse graph datasets, including out-of-distribution and heterophilic graphs, without requiring extensive training or search. Use when the user wants to benchmark on NAS-Bench-Graph & OOD Graphs (Cora, Citeseer, PubMed, CS, Physics, Photo, Computer, ogbn-arXiv, DBLP, Flickr, Actor), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.06717
  bibtex_key: wang2024designgnn
  confidence: high
---

# gnn-design-eval

> Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models — Wang et al. (2024) (arXiv:2408.06717, 2024)

## What this evaluates

Evaluates an LLM-guided framework's ability to automatically propose and refine Graph Neural Network architectures for node classification across diverse graph datasets, including out-of-distribution and heterophilic graphs, without requiring extensive training or search.

## Datasets

- **NAS-Bench-Graph & OOD Graphs (Cora, Citeseer, PubMed, CS, Physics, Photo, Computer, ogbn-arXiv, DBLP, Flickr, Actor)** — total ?; splits: train/val/test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Mean node classification accuracy over 10 independent runs with different random seeds. Standard deviation is also reported alongside the mean.
- `hit_rate` — range: [0, 1]
  - Proportion of times the empirically top-performing benchmark graphs are included in the Top-Ns retrieved benchmarks by the similarity module.
- `Kendall's tau` — range: [-1, 1]
  - Rank correlation coefficient measuring the agreement between predicted task similarities and empirical performance similarities.

## Input / output format

**Input**: Graph dataset properties (e.g., node/edge counts, feature dimensions, label distribution) and optional textual descriptions.

**Output**: A GNN architecture proposal (layer types, connections, hyperparameters) from a predefined unified search space.

## Scoring recipe

```python
def compute_accuracy(proposals, dataset):
    accs = []
    for run in range(10):
        model = propose_architecture(dataset.properties)
        train_and_validate(model, dataset)
        accs.append(model.test_accuracy)
    return np.mean(accs), np.std(accs)

def compute_hit_rate(retrieved_top_n, empirical_top_n):
    return len(set(retrieved_top_n) & set(empirical_top_n)) / len(empirical_top_n)
```

## Common pitfalls

- Data leakage: Unseen datasets must be completely excluded from the knowledge retrieval process and anonymized during evaluation.
- Efficiency metric confusion: The paper uses '# proposal validations' as the basic time unit for LLM API traffic, not wall-clock time or FLOPs.
- Reporting format: Results must report mean accuracy ± standard deviation over exactly 10 runs, not single-run results.

## Evidence (verbatim from paper)

> To ensure reliability, we report the average accuracy and standard deviation over 10 runs. For automated methods, we also analyze the best-so-far accuracy after validating 1-30 model proposals to measure short-run efficiency.

## Citation

```bibtex
@misc{wang2024designgnn,
  title={Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2408.06717}
}
```

- arXiv: 2408.06717