Drugood Eval

---
name: drugood-eval
description: Evaluates the out-of-distribution (OOD) generalization and robustness of graph neural networks and sequence models on molecular binding affinity prediction tasks under various domain shifts and annotation noise levels. Use when the user wants to benchmark on DrugOOD, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2201.09637
  bibtex_key: ji2022drugood
  confidence: high
---

# drugood-eval

> DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations — Ji et al. (2022) (arXiv:2201.09637, 2022)

## What this evaluates

Evaluates the out-of-distribution (OOD) generalization and robustness of graph neural networks and sequence models on molecular binding affinity prediction tasks under various domain shifts and annotation noise levels.

## Datasets

- **DrugOOD** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

## Input / output format

**Input**: Molecular graphs (derived from SMILES) for ligand-binding affinity prediction (LBAP), and protein amino acid sequences for structure-based affinity prediction (SBAP).

**Output**: Predicted binding affinity scores or classification labels, generated via a readout function and MLP layer on top of the backbone encoder.

## Scoring recipe

```python
def compute_auroc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)
```

## Common pitfalls

- Using an in-distribution validation set for early stopping and hyperparameter tuning instead of the specified OOD validation set.
- Assuming domain generalization algorithms will consistently outperform Empirical Risk Minimization (ERM) on molecular graph data, as the paper shows ERM often remains competitive or superior.
- Failing to respect the specific domain split definitions (assay, scaffold, size, protein, protein-family), which dictate the nature of the distribution shift being evaluated.

## Evidence (verbatim from paper)

> We adopt the Area under the ROC Curve (AUROC) to estimate model performance; the higher score is better. In all tables in this paper, we report in parentheses the standard deviation of 3 replications, which measures the variability among replications. All datasets show performance drops due to distribution shift, with substantially better ID performance than OOD performance.

## Citation

```bibtex
@misc{ji2022drugood,
  title={DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations},
  author={Ji et al. (2022)},
  year={2022},
  note={arXiv:2201.09637}
}
```

- arXiv: 2201.09637