Casp 2016 Dti Eval

---
name: casp-2016-dti-eval
description: Evaluates deep learning models for drug-target interaction prediction across four tasks: scoring (affinity correlation), ranking (pose affinity ordering), docking (native pose retrieval from decoys), and screening (true binder identification among random molecules). Probes both regression accuracy and virtual screening generalization. Use when the user wants to benchmark on CASF-2016, CSAR NRC-HiQ, or asks about evaluating this task. Reports Docking power (Top-1 Success Rate), Screening power (Top-1% Success Rate & EF).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2008.12249
  bibtex_key: moon2020pignet
  confidence: high
---

# casp-2016-dti-eval

> PIGNet: A physics-informed deep learning model toward generalized drug-target interaction predictions — Moon et al. (2020) (arXiv:2008.12249, 2020)

## What this evaluates

Evaluates deep learning models for drug-target interaction prediction across four tasks: scoring (affinity correlation), ranking (pose affinity ordering), docking (native pose retrieval from decoys), and screening (true binder identification among random molecules). Probes both regression accuracy and virtual screening generalization.

## Datasets

- **CASF-2016** — total ?; splits: test (-1); repo https://github.com/jaechanglim/DTI_PDBbind.git
- **CSAR NRC-HiQ** — total ?; splits: test (-1)

## Metrics

- `Scoring power (Pearson's R)` — range: [0, 1]
  - Pearson correlation coefficient between predicted and experimental binding affinities.
- `Ranking power (Spearman's ρ)` — range: [0, 1]
  - Spearman's rank-correlation coefficient measuring the ability to correctly rank binding affinities of true binders.
- `Docking power (Top-1 Success Rate)` **(primary)** — range: [0, 1]
  - Percentage of complexes where the native binding pose is ranked within the top 1 candidates among computationally generated decoys.
- `Screening power (Top-1% Success Rate & EF)` **(primary)** — range: [0, 1]
  - Success rate: percentage of targets where the true binder is in the top 1% of candidates. Enhancement Factor (EF): ratio of true binders in the top 1% to the expected ratio if randomly selected.

## Input / output format

**Input**: 3D atomic coordinates and connectivity for protein-ligand complexes (native or computationally generated decoy poses), ligand molecular graph/SMILES, and protein sequence/structure.

**Output**: Predicted binding affinity score, ranked list of ligand poses, or binary/continuous screening score for each candidate molecule.

## Scoring recipe

```python
R = pearsonr(predicted_affinities, experimental_affinities)
rho = spearmanr(predicted_affinities, experimental_affinities)

docking_success = sum(1 for c in complexes if native_pose_rank[c] == 1) / len(complexes)

top_k = max(1, int(0.01 * len(candidates)))
screening_success = sum(1 for t in targets if true_binder_rank[t] <= top_k) / len(targets)
EF = screening_success / 0.01
```

## Common pitfalls

- High scoring power does not guarantee high docking or screening power, as models often overfit to true-binding complexes and fail on decoys.
- Scoring power alone is insufficient for evaluation; models must be tested on computationally generated decoys to assess generalization.
- Enhancement factor and success rate thresholds (e.g., top 1%) must be explicitly stated, as they vary across studies and significantly impact reported performance.

## Evidence (verbatim from paper)

> The scoring power measures a linear correlation of predicted binding affinities and experimental values, calculated by a Pearson's correlation coefficient R. The ranking power measures an ability of a model to correctly rank the binding affinities of true binders of the actual binding pose, calculated by a Spearman's rank-correlation coefficient ρ. These two metrics are designed to assess the model's ability upon the stable-and-precise binding structures. On the other hand, the docking power and the screening power deal with the unnatural structures which are generated computationally. The docking power measures an ability of a model to find out the native binding pose of a ligand among computer-generated decoys, quantified as a success rate within the top N candidates. The screening power measures the ability of a model to identify the specific binding ligand for a given target protein among a set of random molecules, quantified as a success rate and an enhancement factor (EF) within the top α percent of candidates.

## Citation

```bibtex
@misc{moon2020pignet,
  title={PIGNet: A physics-informed deep learning model toward generalized drug-target interaction predictions},
  author={Moon et al. (2020)},
  year={2020},
  note={arXiv:2008.12249}
}
```

- arXiv: 2008.12249