Imdrug Eval

---
name: imdrug-eval
description: Evaluates deep learning models for imbalanced and long-tailed classification and regression in AI-aided drug discovery. It probes model robustness to severe class imbalance, open long-tailed distributions, and out-of-distribution chemical splits across graph, sequence, and fingerprint molecular representations. Use when the user wants to benchmark on HIV, SBAP, USPTO-50K, DrugBank, or asks about evaluating this task. Reports Balanced-Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.07921
  bibtex_key: li2022imdrug
  confidence: high
---

# imdrug-eval

> ImDrug: A Benchmark for Deep Imbalanced Learning in AI-aided Drug Discovery — Lanqing Li et al. (2022) (arXiv:2209.07921, 2022)

## What this evaluates

Evaluates deep learning models for imbalanced and long-tailed classification and regression in AI-aided drug discovery. It probes model robustness to severe class imbalance, open long-tailed distributions, and out-of-distribution chemical splits across graph, sequence, and fingerprint molecular representations.

## Datasets

- **HIV** — total 41127; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
- **SBAP** — total 32140; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
- **USPTO-50K** — total 50036; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
- **DrugBank** — total 191808; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug

## Metrics

- `Balanced-Acc` **(primary)** — range: [0, 1]
  - Macro-average of per-class recall: mean over all classes of (true positives / total actual positives) for each class. Insensitive to label distribution.
- `Balanced-F1` — range: [0, 1]
  - Macro-average of per-class F1 scores: harmonic mean of per-class precision and recall, averaged across all classes.
- `AUROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

## Input / output format

**Input**: Molecular structures encoded as graphs (via DGL), SMILES sequences, or Morgan fingerprints. Tasks support single-instance or multi-instance (bag-level) prediction.

**Output**: Predicted class labels or probabilities for classification tasks; continuous values for regression tasks.

## Scoring recipe

```python
def balanced_accuracy(y_true, y_pred, num_classes):
    recalls = []
    for c in range(num_classes):
        mask = y_true == c
        if mask.sum() > 0:
            recalls.append((y_pred[mask] == y_true[mask]).mean())
        else:
            recalls.append(0.0)
    return np.mean(recalls)

def balanced_f1(y_true, y_pred, num_classes):
    f1s = []
    for c in range(num_classes):
        mask = y_true == c
        if mask.sum() > 0:
            tp = (y_pred[mask] == y_true[mask]).sum()
            fp = (y_pred[mask] != y_true[mask]).sum()
            fn = (y_pred[y_true == c] != y_true[mask]).sum()
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
            f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    return np.mean(f1s)
```

## Common pitfalls

- Relying solely on AUROC, which the paper notes is over-optimistic and insensitive to baseline differences in highly imbalanced drug discovery settings.
- Using standard random splits instead of the proposed chemically realistic 'standard' splits, which masks real-world out-of-distribution generalization challenges.
- Ignoring multi-instance prediction settings (e.g., SBAP, DrugBank) where bag-level labels require specialized evaluation rather than instance-level averaging.

## Evidence (verbatim from paper)

> In experiments, we benchmark 11 baselines for conventional imbalanced & long-tailed classification (Sec.[3.3.2]), as well as 5 additional baselines tailored for open LT and imbalanced regression, by reporting the 2 proposed balanced accuracy and balanced F1 measures along with the conventional AUROC. Other metrics for imbalanced learning such as AUPRC and Weighted-F1 are also reported in Appendix H. For hybrid prediction tasks/datasets (Table[II]), unless otherwise specified, the training mode is single-instance prediction by default. All average performance with standard deviations is evaluated over 3 random seeds.

## Citation

```bibtex
@misc{li2022imdrug,
  title={ImDrug: A Benchmark for Deep Imbalanced Learning in AI-aided Drug Discovery},
  author={Lanqing Li et al. (2022)},
  year={2022},
  note={arXiv:2209.07921}
}
```

- arXiv: 2209.07921