Dti Benchmark Eval

---
name: dti-benchmark-eval
description: Evaluates the ability of molecular models to predict drug-target interactions (DTI) by classifying whether a given drug and protein target pair binds. It probes the model's capacity to integrate diverse molecular representations (sequences, graphs, structures) and interaction layers to distinguish positive binding pairs from negative ones. Use when the user wants to benchmark on Davis, BIOSNAP, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15010
  bibtex_key: liu2024flexmol
  confidence: high
---

# dti-benchmark-eval

> FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learning — Liu et al. (2024) (arXiv:2410.15010, 2024)

## What this evaluates

Evaluates the ability of molecular models to predict drug-target interactions (DTI) by classifying whether a given drug and protein target pair binds. It probes the model's capacity to integrate diverse molecular representations (sequences, graphs, structures) and interaction layers to distinguish positive binding pairs from negative ones.

## Datasets

- **Davis** — total ?; splits: train (-1), val (-1), test (-1)
- **BIOSNAP** — total 13741; splits: train (-1), val (-1), test (-1)

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `PR-AUC` — range: [0, 1]
  - Area under the Precision-Recall curve, measuring the trade-off between precision and recall across thresholds, particularly sensitive to class imbalance.

## Input / output format

**Input**: Paired drug and protein molecular data, which can include drug sequences (ds), drug graphs (dg), protein sequences (ps), and protein graphs (pg) depending on the encoder configuration.

**Output**: Binary interaction probability or score indicating the likelihood of binding between the drug and protein pair.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    roc_auc = roc_auc_score(y_true, y_pred)
    pr_auc = average_precision_score(y_true, y_pred)
    return {'roc-auc': roc_auc, 'pr-auc': pr_auc}
# y_true: 1 for positive (Kd<30 or known binding), 0 for negative (sampled unseen pairs)
# Results are averaged over 5 random 7:2:1 train/val/test splits.
```

## Common pitfalls

- Negative samples are generated by sampling unseen pairs rather than random negative sampling, which may inflate performance compared to standard benchmarks.
- A random 7:2:1 train/val/test split is used instead of target-independent or scaffold splits, potentially causing data leakage and overly optimistic generalization estimates.

## Evidence (verbatim from paper)

> For both the DAVIS and BIOSNAP datasets, we conducted a random split in the ratio of 7:2:1 for training, validation, and testing, respectively. Each test was repeated five times to mitigate any randomness, and the average results were computed. Table 4 presents the performance metrics of the selected baseline models and the FlexMol models, including ROC-AUC (Receiver Operating Characteristic - Area Under the Curve) and PR-AUC (Precision-Recall Area Under the Curve).

## Citation

```bibtex
@misc{liu2024flexmol,
  title={FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learning},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.15010}
}
```

- arXiv: 2410.15010