Protein Ligand Binding Eval

---
name: protein-ligand-binding-eval
description: Evaluates a model's ability to predict protein-ligand binding using only sequence data. It probes generalization across diverse protein targets, novel chemical scaffolds, and external benchmarks by measuring ranking performance between binders and decoys. Use when the user wants to benchmark on DEL Protein Split, DEL Chemical Library Split, MF-PCBA, Public Binders/Decoys, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13503
  bibtex_key: kleinsasser2026hermes
  confidence: high
---

# protein-ligand-binding-eval

> Hermes: Large DEL Datasets Train Generalizable Protein-Ligand Binding Prediction Models — Kleinsasser et al. (2026) (arXiv:2602.13503, 2026)

## What this evaluates

Evaluates a model's ability to predict protein-ligand binding using only sequence data. It probes generalization across diverse protein targets, novel chemical scaffolds, and external benchmarks by measuring ranking performance between binders and decoys.

## Datasets

- **DEL Protein Split** — total ?; splits: test (-1)
- **DEL Chemical Library Split** — total ?; splits: test (-1)
- **MF-PCBA** — total ?; splits: test (-1)
- **Public Binders/Decoys** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen binder is ranked higher than a randomly chosen non-binder/decoy.

## Input / output format

**Input**: Protein amino acid sequence and ligand molecular representation (SMILES/sequence), processed via pre-trained embeddings (ESM2 and ChemBERTa) with cross-attention.

**Output**: Binary binding prediction or continuous binding probability score.

## Scoring recipe

```python
auroc_scores = []
for target in targets:
    preds = get_predictions(target)
    labels = get_labels(target)
    auroc_scores.append(roc_auc_score(labels, preds))
mean_auroc = np.mean(auroc_scores)
std_auroc = np.std(auroc_scores)
```

## Common pitfalls

- N in evaluation tables refers to the number of protein targets, not the number of ligand pairs.
- Some benchmarks (e.g., MF-PCBA) may contain data overlapping with competitor models' training sets, inflating apparent performance.
- Datasets are often subsampled (e.g., to 50k samples) for inference feasibility, which may not reflect full-dataset evaluation conditions.

## Evidence (verbatim from paper)

> Table 3: Hermes vs benchmarks per-protein AUROC comparison. AUROC columns indicate mean + standard deviation AUROC scores across all targets in the evaluation set.

## Citation

```bibtex
@misc{kleinsasser2026hermes,
  title={Hermes: Large DEL Datasets Train Generalizable Protein-Ligand Binding Prediction Models},
  author={Kleinsasser et al. (2026)},
  year={2026},
  note={arXiv:2602.13503}
}
```

- arXiv: 2602.13503