Dti Prediction Eval

---
name: dti-prediction-eval
description: This benchmark evaluates a model's ability to predict drug-target interactions by integrating molecular graphs and protein sequences into a heterogeneous interaction network. It probes the model's capacity to learn hierarchical graph representations and distinguish interacting from non-interacting drug-protein pairs. Use when the user wants to benchmark on DTI Benchmark, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.11757
  bibtex_key: song2025gig
  confidence: high
---

# dti-prediction-eval

> A Graph-in-Graph Learning Framework for Drug-Target Interaction Prediction — Yuehua Song, Yong Gao (2025) (arXiv:2507.11757, 2025)

## What this evaluates

This benchmark evaluates a model's ability to predict drug-target interactions by integrating molecular graphs and protein sequences into a heterogeneous interaction network. It probes the model's capacity to learn hierarchical graph representations and distinguish interacting from non-interacting drug-protein pairs.

## Datasets

- **DTI Benchmark** — total 1923; splits: (unstated)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve, plotting True Positive Rate against False Positive Rate across classification thresholds. Higher values indicate better global separability.
- `AUPRC` — range: [0, 1]
  - Area under the precision-recall curve, evaluating the trade-off between precision and recall across thresholds. Particularly meaningful for sparse interaction datasets.
- `F1 Score` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated at a fixed 0.5 decision threshold. Reflects the classifier's discriminative power under balanced sampling.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient, measuring the correlation between observed and predicted binary classifications. Provides a nuanced view of predictive consistency across both positive and negative samples.

## Input / output format

**Input**: Drug and target nodes represented as molecular graphs (derived from SMILES) and protein sequences, embedded as meta-nodes in a bipartite drug-target interaction graph. Pairs are constructed based on known interaction edges.

**Output**: A scalar probability score indicating the likelihood of interaction for each drug-target pair.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, f1_score, matthews_corrcoef

def compute_metrics(y_true, y_pred_prob):
    auc = roc_auc_score(y_true, y_pred_prob)
    auprc = average_precision_score(y_true, y_pred_prob)
    y_pred = (y_pred_prob >= 0.5).astype(int)
    f1 = f1_score(y_true, y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)
    return {'AUC': auc, 'AUPRC': auprc, 'F1': f1, 'MCC': mcc}
```

## Common pitfalls

- Training uses hard negative sampling with a 1:1 positive-to-negative ratio, which can artificially inflate F1 and AUPRC if the test set retains this balance rather than reflecting real-world sparsity.
- The dataset is described as a 'novel benchmark' with no public release link or HuggingFace ID provided, making independent replication difficult.
- AUC evaluates ranking robustness and is less affected by the hard negative sampling strategy, whereas AUPRC and F1 are highly sensitive to the positive class prevalence in the evaluation split.

## Evidence (verbatim from paper)

> To evaluate the performance of the models, we selected four key metrics: the area under the receiver operating characteristic curve (AUC) [58], the area under the precision-recall curve (AUPRC) [59, 61], the F1 Score [60], and the Matthews Correlation Coefficient (MCC) [62]. These metrics were chosen to provide a comprehensive view of the model's predictive capabilities, especially considering the use of hard negative sampling during the training phase.

## Citation

```bibtex
@misc{song2025gig,
  title={A Graph-in-Graph Learning Framework for Drug-Target Interaction Prediction},
  author={Yuehua Song, Yong Gao (2025)},
  year={2025},
  note={arXiv:2507.11757}
}
```

- arXiv: 2507.11757