Crossdocked Sbdd Eval

---
name: crossdocked-sbdd-eval
description: Evaluates a model's ability to generate novel, drug-like molecules with high binding affinity for unseen protein pockets in structure-based drug design. It probes the trade-offs between binding energy, molecular properties, and synthesis feasibility. Use when the user wants to benchmark on CrossDocked-100k, or asks about evaluating this task. Reports Vina Dock.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.03223
  bibtex_key: shen2023tacogfn
  confidence: high
---

# crossdocked-sbdd-eval

> TacoGFN: Target-conditioned GFlowNet for Structure-based Drug Design — Shen et al. (2023) (arXiv:2310.03223, 2023)

## What this evaluates

Evaluates a model's ability to generate novel, drug-like molecules with high binding affinity for unseen protein pockets in structure-based drug design. It probes the trade-offs between binding energy, molecular properties, and synthesis feasibility.

## Datasets

- **CrossDocked-100k** — total 100000; splits: train (-1), test (100)

## Metrics

- `Validity` — range: percent
  - Percentage of unique generated molecules that are free of reconstruction errors and disconnections, validated using RDKit.
- `Vina Dock` **(primary)** — range: other
  - Approximates the binding energy between a generated molecule and a protein pocket using AutoDock Vina; lower scores indicate higher binding affinity.
- `High Affinity` — range: percent
  - Percentage of generated molecules that achieve a higher binding affinity (lower Vina Dock score) than the reference/native molecule.
- `QED` — range: [0, 1]
  - Quantitative Estimate of Drug-likeness; estimates a molecule's suitability as an oral drug based on physicochemical properties.
- `SA` — range: [0, 1]
  - Synthetic Accessibility score estimating synthesis difficulty, normalized to [0, 1] using the formula (10 - SA)/9.
- `Diversity` — range: [0, 1]
  - Average pairwise Tanimoto distance calculated from molecular fingerprints of the generated set.
- `Success Rate` — range: percent
  - Percentage of generated molecules that simultaneously satisfy QED > 0.25, SA > 0.59, and Vina Dock < -8.18.
- `Time` — range: other
  - Average runtime in seconds required to generate 100 unique and valid molecules for a single protein pocket.

## Input / output format

**Input**: 3D protein pocket structure/coordinates

**Output**: 100 generated molecule structures (ligands) per protein pocket

## Scoring recipe

```python
def compute_metrics(predictions, pocket, ref_dock):
    valid_mols = [m for m in predictions if rdkit.is_valid(m)]
    validity = len(valid_mols) / len(predictions)
    vina_scores = [dock_vina(m, pocket) for m in valid_mols]
    high_aff = sum(1 for s in vina_scores if s < ref_dock) / len(valid_mols)
    qed_vals = [calc_qed(m) for m in valid_mols]
    sa_vals = [(10 - calc_sa(m)) / 9 for m in valid_mols]
    diversity = avg_tanimoto_distance(valid_mols)
    success = sum(1 for m in valid_mols if calc_qed(m) > 0.25 and calc_sa(m) > 0.59 and dock_vina(m, pocket) < -8.18) / len(valid_mols)
    return validity, vina_scores, high_aff, qed_vals, sa_vals, diversity, success
```

## Common pitfalls

- Heavy molecules often achieve better Vina Dock scores but violate drug-likeness (QED) and Rule of 5 constraints, creating a trade-off that skews naive docking optimization.
- Success rate thresholds (QED > 0.25, SA > 0.59, Vina Dock < -8.18) are strict and specific to this benchmark; relaxing them significantly inflates reported performance.
- Diversity and reward quality trade off based on sampling temperature β; higher β yields better affinity but lower diversity, requiring careful temperature selection for fair comparison.

## Evidence (verbatim from paper)

> In all evaluations, each structure-based generative model is tasked to produce 100 molecules (ligands) for each of the 100 unseen protein pockets from the CrossDock-100k test set. Evaluation metrics. We adopt the following commonly used metrics from Guan et al. (2023a) and Reidenbach (2024): (1) Validity is the percentage of unique generated molecules free of reconstruction errors and disconnections as determined by RDKit. (2) Vina Dock approximates the binding energy between a generated molecule and a protein pocket, where a lower docking score indicates a higher binding affinity. (7) Success Rate is the percentage of molecules which pass the same criteria (QED > 0.25, SA > 0.59, Vina Dock < -8.18) as in Long et al. (2022); Guan et al. (2023b); Zhou et al. (2024); Reidenbach (2024).

## Citation

```bibtex
@misc{shen2023tacogfn,
  title={TacoGFN: Target-conditioned GFlowNet for Structure-based Drug Design},
  author={Shen et al. (2023)},
  year={2023},
  note={arXiv:2310.03223}
}
```

- arXiv: 2310.03223