Gnn Vs Dnn Lhc Bb Eval

---
name: gnn-vs-dnn-lhc-bb-eval
description: Compares graph neural networks against deep fully-connected feedforward networks for binary classification of b-quark pairs in top-quark-antiquark collisions at the LHC. It probes whether explicit relational inductive biases in GNNs outperform permutation-invariant DNNs when provided with equivalent kinematic and relational features. Use when the user wants to benchmark on LHC t\bar{t} b\bar{b} event classification, or asks about evaluating this task. Reports mean ROC-AUC ($\mu_{\mathrm{AUC}}$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.17386
  bibtex_key: pfeffer2024gnntestbench
  confidence: high
---

# gnn-vs-dnn-lhc-bb-eval

> A case study of sending graph neural networks back to the test bench for applications in high-energy particle physics — Emanuel Pfeffer et al. (arXiv:2402.17386, 2024)

## What this evaluates

Compares graph neural networks against deep fully-connected feedforward networks for binary classification of b-quark pairs in top-quark-antiquark collisions at the LHC. It probes whether explicit relational inductive biases in GNNs outperform permutation-invariant DNNs when provided with equivalent kinematic and relational features.

## Datasets

- **LHC t\bar{t} b\bar{b} event classification** — total ?; splits: train (-1)

## Metrics

- `mean ROC-AUC ($\mu_{\mathrm{AUC}}$)` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Computed as the average ROC-AUC over an ensemble of 10 independent training repetitions. Uncertainty $\Delta \mu_{\mathrm{AUC}}$ is derived from the sample variance of the ensemble.

## Input / output format

**Input**: Kinematic feature vectors for reconstructed particles/jets (e.g., $\phi$, $\eta$, energy). For GNNs, these are node features on a fully connected graph with edge weights $\omega_{ij}$ encoding relational information (e.g., $\Delta R$, $\Delta R^{-1}$, or invariant mass $m$). DNNs receive flattened feature vectors $\mathbf{x}^{\mathrm{DNN}}$ or reduced versions $\mathbf{x}_{\mathrm{red}}^{\mathrm{DNN}}$.

**Output**: Binary classification probability scores for signal vs. background events, used to compute ROC-AUC.

## Scoring recipe

```python
auc_scores = []
for _ in range(10):
    model = train_model(data, architecture)
    auc_scores.append(compute_roc_auc(model.predictions, gold_labels))
mean_auc = np.mean(auc_scores)
uncertainty = np.std(auc_scores)
# Outlier removal: remove runs where auc > mean + 1.5*std if it changes std by >= 0.0025
```

## Common pitfalls

- GNNs perform worse than DNNs if edge weights are random or unweighted; domain-knowledge-driven relational features are required to unlock GNN advantages.
- Performance is evaluated via ensemble averaging (10 runs) rather than a single train/test split, making variance $\Delta \mu_{\mathrm{AUC}}$ a critical reporting metric.
- DNNs implicitly capture relational information through positional encoding of parton associations, which can narrow the performance gap with GNNs.

## Evidence (verbatim from paper)

> The metric by which to judge the success of an NN to fulfill the task is chosen to be the mean of the ROC-AUC $\mu_{\mathrm{AUC}}$ based on the training setup, as described in Section 2.4. The bars associated with the points indicate the uncertainty $\Delta \mu_{\mathrm{AUC}}$ in $\mu_{\mathrm{AUC}}$ due to random choices in the training, as described in Section 2.4.

## Citation

```bibtex
@misc{pfeffer2024gnntestbench,
  title={A case study of sending graph neural networks back to the test bench for applications in high-energy particle physics},
  author={Emanuel Pfeffer et al.},
  year={2024},
  note={arXiv:2402.17386}
}
```

- arXiv: 2402.17386