Neurokg Eval

---
name: neurokg-eval
description: Evaluates a heterogeneous graph transformer's ability to learn multi-scale biological relationships and predict missing links in a brain knowledge graph. It probes downstream capabilities including genome-wide screen enrichment, pesticide toxicity ranking, and drug repurposing forecasting across neurological diseases. Use when the user wants to benchmark on NeuroKG, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.13724
  bibtex_key: noori2025proton
  confidence: high
---

# neurokg-eval

> Graph AI generates neurological hypotheses validated in molecular, organoid, and clinical systems — Noori et al. (2025) (arXiv:2512.13724, 2025)

## What this evaluates

Evaluates a heterogeneous graph transformer's ability to learn multi-scale biological relationships and predict missing links in a brain knowledge graph. It probes downstream capabilities including genome-wide screen enrichment, pesticide toxicity ranking, and drug repurposing forecasting across neurological diseases.

## Datasets

- **NeuroKG** — total 147020; splits: held-out test (-1); repo https://github.com/mims-harvard/PROTON

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `Normalized Enrichment Score (NES)` — range: other
  - A Kolmogorov-Smirnov statistic adjusted for multiple hypothesis testing and gene set size, used in GSEA to quantify overrepresentation of experimental hits in predicted rankings.
- `macro-averaged recall at k (R@k)` — range: percent
  - The average percentage of known relevant drugs recovered within the top k predictions across all evaluated diseases.

## Input / output format

**Input**: Heterogeneous graph containing 147,020 nodes across 16 types (genes, proteins, cell types, brain regions, phenotypes, drugs) and 7,366,745 edges across 47 types. Tasks involve node pairs for link prediction, disease/compound nodes with graph context for downstream screens, and fine-tuning datasets of labeled pesticides.

**Output**: Continuous link prediction scores or ranks for node pairs; ranked lists of genes/proteins/compounds; or binary toxicity predictions for pesticides.

## Scoring recipe

```python
def compute_auroc(y_true, y_scores):
    return roc_auc_score(y_true, y_scores)

def compute_recall_at_k(relevant_items, ranked_list, k):
    top_k = ranked_list[:k]
    return len(set(relevant_items) & set(top_k)) / len(relevant_items)

def compute_nes(experimental_hits, predicted_rankings):
    return gsea_normalized_enrichment_score(experimental_hits, predicted_rankings)
```

## Common pitfalls

- Information leakage occurs if drug-disease edges for the target disease or related diseases are not removed from the graph before training disease-specific splits.
- Genes or proteins with direct links to the query node (e.g., PD or α-synuclein) must be excluded from enrichment analyses to prevent tautological results.
- List-size differences in GWAS hit sets must be controlled via random subsampling to ensure robust statistical comparisons across diseases.

## Evidence (verbatim from paper)

> Proton was trained on the NeuroKG dataset using a self-supervised link prediction objective (Methods Sec.[2.2]). Through Bayesian hyperparameter optimization, we selected a model architecture that achieved high link prediction performance (AUROC =0.9145; accuracy =82.23%) on the held-out test set.

## Citation

```bibtex
@misc{noori2025proton,
  title={Graph AI generates neurological hypotheses validated in molecular, organoid, and clinical systems},
  author={Noori et al. (2025)},
  year={2025},
  note={arXiv:2512.13724}
}
```

- arXiv: 2512.13724