Mudi Eval

---
name: mudi-eval
description: This benchmark evaluates a model's ability to predict pharmacodynamic drug-drug interactions (Synergism, Antagonism, or New Effect) using multimodal inputs including text, chemical formulas, molecular graphs, and images. It specifically probes cross-modal reasoning and generalization to unseen drug pairs under both direction-aware and direction-agnostic matching settings. Use when the user wants to benchmark on MUDI, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01478
  bibtex_key: ngo2025mudi
  confidence: high
---

# mudi-eval

> MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions — Ngo et al. (2025) (arXiv:2506.01478, 2025)

## What this evaluates

This benchmark evaluates a model's ability to predict pharmacodynamic drug-drug interactions (Synergism, Antagonism, or New Effect) using multimodal inputs including text, chemical formulas, molecular graphs, and images. It specifically probes cross-modal reasoning and generalization to unseen drug pairs under both direction-aware and direction-agnostic matching settings.

## Datasets

- **MUDI** — total 310532; splits: test (-1); repo https://github.com/hoangbros03/MUDI

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: 2 * (P * R) / (P + R). Computed per positive class (Synergism, Antagonism, New Effect) and aggregated via Micro or Macro averaging. Negative examples (No Interaction) are excluded from metric computation.

## Input / output format

**Input**: A pair of drugs represented by multimodal features: textual fields (name, description, SMILES, formula), molecular structure graphs, and structure images.

**Output**: A single categorical label: Synergism, Antagonism, or New Effect.

## Scoring recipe

```python
positive_classes = ['Synergism', 'Antagonism', 'New Effect']
f1_scores = []
for c in positive_classes:
    tp = sum(p == c and g == c for p, g in zip(preds, gold))
    fp = sum(p == c and g != c for p, g in zip(preds, gold))
    fn = sum(p != c and g == c for p, g in zip(preds, gold))
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_scores.append(2 * p * r / (p + r) if (p + r) > 0 else 0)
micro_f1 = compute_micro_f1(preds, gold, positive_classes)
macro_f1 = mean(f1_scores)
```

## Common pitfalls

- Negative examples ('No Interaction') are used for training but must be excluded when computing evaluation metrics.
- Evaluation must explicitly state directionality: direction-aware matching treats (A, B) and (B, A) as distinct, while direction-agnostic ignores order, causing large performance gaps.
- Severe class imbalance exists for the 'New Effect' label, which drastically lowers per-class F1 scores and requires careful macro-averaging interpretation.

## Evidence (verbatim from paper)

> Evaluation Metrics. We report Precision, Recall, and F1 scores for each of the three positive classes, along with Micro and Macro averages. Negative examples (No Interaction) are included during training but excluded from positive-class metric computation, following common biomedical evaluation practice [21]. Evaluation Settings. We evaluate under two settings: (i) direction-aware matching, where (DRUG1, DRUG2) ≠ (DRUG2, DRUG1), and (ii) direction-agnostic matching, where order is ignored.

## Citation

```bibtex
@misc{ngo2025mudi,
  title={MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions},
  author={Ngo et al. (2025)},
  year={2025},
  note={arXiv:2506.01478}
}
```

- arXiv: 2506.01478