Pdbbind Low Similarity Eval

---
name: pdbbind-low-similarity-eval
description: Evaluates how well 3D binding affinity models generalise to unseen proteins and novel ligands in low-data regimes. It uses a strict low-Tanimoto-similarity split of the PDBBind dataset to prevent data leakage and benchmark generalisation capabilities. Use when the user wants to benchmark on PDBBind, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.12995
  bibtex_key: buhmann2024lowsimpdbbind
  confidence: medium
---

# pdbbind-low-similarity-eval

> Improving generalisability of 3D binding affinity models in low data regimes — Buhmann et al. (2024) (arXiv:2409.12995, 2024)

## What this evaluates

Evaluates how well 3D binding affinity models generalise to unseen proteins and novel ligands in low-data regimes. It uses a strict low-Tanimoto-similarity split of the PDBBind dataset to prevent data leakage and benchmark generalisation capabilities.

## Datasets

- **PDBBind** — total ?; splits: train (-1), test (-1); repo https://github.com/Exscientia/low-sim-pdbbind

## Metrics

- `performance` **(primary)** — range: other
  - Standard regression metric for binding affinity prediction (typically RMSE or MAE between predicted and experimental values). The paper refers to it generally as 'performance'.

## Input / output format

**Input**: 3D coordinates and atom types for protein pockets and ligands, optionally with explicitly added hydrogen atoms.

**Output**: Continuous binding affinity value (e.g., pKd, pKi, or ΔG).

## Scoring recipe

```python
def score(predictions, gold):
    rmse = np.sqrt(np.mean((predictions - gold) ** 2))
    mae = np.mean(np.abs(predictions - gold))
    return {'rmse': rmse, 'mae': mae}
```

## Common pitfalls

- Data leakage if train/test splits are based on high structural similarity rather than low Tanimoto similarity.
- Ignoring hydrogen atoms in protein structures, which significantly hurts performance in low-data regimes.
- Using single-protein local models that fail to generalise to unseen proteins compared to global 3D models.

## Evidence (verbatim from paper)

> The results reveal a clear advantage for global 3D models over local models in low data regimes. ... As seen in Figure 3, the performance of the global 3D models is relatively consistent with the level of training data (improving slightly) whereas the Single-Protein local model suffers greatly at low data, dramatically improving with increasing amounts of training data.

## Citation

```bibtex
@misc{buhmann2024lowsimpdbbind,
  title={Improving generalisability of 3D binding affinity models in low data regimes},
  author={Buhmann et al. (2024)},
  year={2024},
  note={arXiv:2409.12995}
}
```

- arXiv: 2409.12995