Fedgraphnn System Eval

---
name: fedgraphnn-system-eval
description: Evaluates the computational efficiency and security overhead of federated graph neural network training. Measures how system-level metrics like training time, FLOPs, and parameter counts scale across diverse graph datasets under non-IID data partitioning and secure aggregation protocols. Use when the user wants to benchmark on SIDER, BACE, Clintox, BBBP, Tox21, FreeSolv, ESOL, Lipo, hERG, QM9, Ciao, Epinions, CORA, Citeseer, DBLP, PubMed, or asks about evaluating this task. Reports Wall-clock Time.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.07145
  bibtex_key: he2021fedgraphnn
  confidence: high
---

# fedgraphnn-system-eval

> FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks — He et al. (2021) (arXiv:2104.07145, 2021)

## What this evaluates

Evaluates the computational efficiency and security overhead of federated graph neural network training. Measures how system-level metrics like training time, FLOPs, and parameter counts scale across diverse graph datasets under non-IID data partitioning and secure aggregation protocols.

## Datasets

- **SIDER** — total ?; splits: train (-1)
- **BACE** — total ?; splits: train (-1)
- **Clintox** — total ?; splits: train (-1)
- **BBBP** — total ?; splits: train (-1)
- **Tox21** — total ?; splits: train (-1)
- **FreeSolv** — total ?; splits: train (-1)
- **ESOL** — total ?; splits: train (-1)
- **Lipo** — total ?; splits: train (-1)
- **hERG** — total ?; splits: train (-1)
- **QM9** — total ?; splits: train (-1)
- **Ciao** — total ?; splits: train (-1)
- **Epinions** — total ?; splits: train (-1)
- **CORA** — total ?; splits: train (-1)
- **Citeseer** — total ?; splits: train (-1)
- **DBLP** — total ?; splits: train (-1)
- **PubMed** — total ?; splits: train (-1)

## Metrics

- `Wall-clock Time` **(primary)** — range: seconds
  - Total elapsed time for federated training across all clients and rounds, measured in seconds, minutes, or hours.
- `Average FLOP` — range: FLOPs
  - Mean floating-point operations performed per client during local training, measured in thousands or millions of FLOPs.
- `Parameters` — range: count
  - Total number of trainable parameters in the GNN model (GCN, GAT, or GraphSAGE).

## Input / output format

**Input**: Graph datasets partitioned non-IID across multiple clients. Each client receives a local subgraph and trains a GNN (GCN, GAT, or GraphSAGE) using FedAvg or LightSecAgg aggregation.

**Output**: System performance logs reporting wall-clock training time, average FLOPs per client, and model parameter counts. For security evaluation, total running time of secure aggregation protocols.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # System metrics are logged directly by the FedGraphNN profiler, not derived from predictions/gold
    wall_clock_time = log_wall_clock_time()
    avg_flops = log_average_flops()
    param_count = count_parameters()
    return {'wall_clock_time': wall_clock_time, 'avg_flops': avg_flops, 'params': param_count}
```

## Common pitfalls

- Confusing system efficiency metrics (time, FLOPs) with model predictive performance (accuracy, F1).
- Assuming standard IID data splits; the benchmark explicitly uses realistic non-IID partitioning.
- Comparing wall-clock times across different hardware configurations without normalization, as the benchmark specifies 8x NVIDIA Quadro RTX 5000 GPUs.

## Evidence (verbatim from paper)

> Table 6: System-Level Performance Metrics for Graph-Level FedGraphNN tasks with FedAvg (Hardware: 8 x NVIDIA Quadro RTX 5000 GPU (16GB/GPU); RAM: 512G; CPU: Intel Xeon Gold 5220R 2.20GHz). ... Wall-clock Time ... Average FLOP ... Parameters

## Citation

```bibtex
@misc{he2021fedgraphnn,
  title={FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks},
  author={He et al. (2021)},
  year={2021},
  note={arXiv:2104.07145}
}
```

- arXiv: 2104.07145