Synthetic Drug Data Eval

---
name: synthetic-drug-data-eval
description: Evaluates the distributional fidelity of generated pharmacokinetic and drug-target interaction properties against real data, and measures the utility of the synthetic data for downstream regression tasks. Use when the user wants to benchmark on TDCommons/BindingDB PK & DTI Collection, or asks about evaluating this task. Reports Hellinger Distance (HD).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.09837
  bibtex_key: hu2025domainknowledge
  confidence: high
---

# synthetic-drug-data-eval

> Domain Knowledge Infused Conditional Generative Models for Accelerating Drug Discovery — Bing Hu et al. (arXiv:2510.09837, 2025)

## What this evaluates

Evaluates the distributional fidelity of generated pharmacokinetic and drug-target interaction properties against real data, and measures the utility of the synthetic data for downstream regression tasks.

## Datasets

- **TDCommons/BindingDB PK & DTI Collection** — total ?; splits: train (-1), test (-1); repo https://github.com/GenerativeDrugDiscovery/xImagand-DKI

## Metrics

- `Hellinger Distance (HD)` **(primary)** — range: [0, 1]
  - Quantifies similarity between two discrete probability distributions P and Q. Formula: HD^2(p,q) = 1/2 * sum((sqrt(p_i) - sqrt(q_i))^2). Lower values indicate closer distribution match.
- `Differential Pairwise Correlations (DPC)` — range: [0, 2]
  - Measures bivariate correlation similarity between real and synthetic data. Formula: |rho_XY_r - rho_XY_s|, where rho is the Pearson correlation coefficient. Values near 0 indicate high similarity.
- `Mean Squared Error (mse)` — range: [0, inf)
  - Standard regression loss measuring average squared difference between predicted and actual values.
- `R-Squared (R2)` — range: (-inf, 1]
  - Proportion of variance in the dependent variable predictable from the independent variables.
- `Pearson Correlation Coefficient (pcc)` — range: [-1, 1]
  - Linear correlation between predicted and actual values.

## Input / output format

**Input**: SMILES strings and protein embeddings (T5 and ProtBERT) conditioned on domain knowledge (GO-derived protein embeddings and molecular fingerprints).

**Output**: Continuous pharmacokinetic (PK) and drug-target interaction (DTI) property values (regression targets).

## Scoring recipe

```python
def compute_hd(real_dist, synth_dist):
    return 0.5 * sum((sqrt(p) - sqrt(q))**2 for p, q in zip(real_dist, synth_dist))

def compute_dpc(real_data, synth_data, x, y):
    rho_r = pearsonr(real_data[x], real_data[y])
    rho_s = pearsonr(synth_data[x], synth_data[y])
    return abs(rho_r - rho_s)

def compute_mle(real_train, real_test, synth_train, synth_test):
    model_r = LinearRegression().fit(real_train, real_train_y)
    model_s = LinearRegression().fit(synth_train, synth_train_y)
    pred_r = model_r.predict(real_test)
    pred_s = model_s.predict(synth_test)
    return mse(real_test_y, pred_r), r2(real_test_y, pred_r), pearsonr(real_test_y, pred_r)
```

## Common pitfalls

- HD requires discretization/binning of continuous PK/DTI properties before distribution comparison.
- DPC omits pairwise combinations with fewer than 10 examples, potentially biasing correlation estimates.
- MLE evaluation uses a specific 50/50 real and 90/10 synthetic split with 1.5 IQR outlier removal, which may not generalize to other splits.

## Evidence (verbatim from paper)

> Hellinger distance (HD) quantifies the similarity between two probability distributions and can be used as a summary statistic of differences for each PK target property between real and synthetic datasets. Given two discrete probability distributions $P\={p_{1},p_{2},...,p_{n}}$ and $Q\={q_{1},q_{2},...,q_{n}}$, the HD between $P$ and $Q$ is expressed in Equation [5].

## Citation

```bibtex
@misc{hu2025domainknowledge,
  title={Domain Knowledge Infused Conditional Generative Models for Accelerating Drug Discovery},
  author={Bing Hu et al.},
  year={2025},
  note={arXiv:2510.09837}
}
```

- arXiv: 2510.09837