Molecular Ue Eval

---
name: molecular-ue-eval
description: This benchmark evaluates uncertainty estimation methods for molecular force fields by measuring predictive accuracy on equilibrium structures, calibration of uncertainty scores, and out-of-distribution detection capabilities on non-equilibrium or left-out molecular configurations. Use when the user wants to benchmark on MD17, QM7X, or asks about evaluating this task. Reports AUC-ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.14916
  bibtex_key: wollschlaeger2023uncertainty
  confidence: high
---

# molecular-ue-eval

> Uncertainty Estimation for Molecules: Desiderata and Methods — Wollschläger et al. (2023) (arXiv:2306.14916, 2023)

## What this evaluates

This benchmark evaluates uncertainty estimation methods for molecular force fields by measuring predictive accuracy on equilibrium structures, calibration of uncertainty scores, and out-of-distribution detection capabilities on non-equilibrium or left-out molecular configurations.

## Datasets

- **MD17** — total ?; splits: train (-1), test (-1)
- **QM7X** — total ?; splits: train (-1), test (-1)

## Metrics

- `MAE` — range: other
  - Mean Absolute Error computed separately for energy and forces. Formula: (1/N) * sum(|y_pred - y_true|). Units vary by dataset (kcal/mol or eV for energy; kcal/(mol·Å) or eV/Å for forces).
- `Calibration Score` — range: other
  - Regression calibration score from Charpentier et al. (2022) measuring how well predicted uncertainties align with actual prediction errors.
- `AUC-ROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve computed on uncertainty scores to separate in-distribution from out-of-distribution samples. A value closer to 1 indicates better separation.

## Input / output format

**Input**: Molecular structures represented as atom types and 3D coordinates (equilibrium or non-equilibrium configurations).

**Output**: Predicted total energy, atomic forces, and associated uncertainty estimates (e.g., variance or predictive log-likelihood).

## Scoring recipe

```python
def compute_metrics(predictions, gold, uncertainty_scores, labels):
    # MAE for energy and forces
    mae_energy = np.mean(np.abs(predictions['energy'] - gold['energy']))
    mae_force = np.mean(np.abs(predictions['force'] - gold['force']))
    # AUC-ROC for OOD detection
    auc = roc_auc_score(labels, uncertainty_scores)
    # Calibration score (Charpentier et al. 2022)
    cal = compute_calibration_score(predictions, gold, uncertainty_scores)
    return {'MAE_energy': mae_energy, 'MAE_force': mae_force, 'AUC-ROC': auc, 'Calibration': cal}
```

## Common pitfalls

- MCD dropout sacrifices predictive accuracy for uncertainty estimates, with energy accuracy deteriorating significantly at higher dropout rates.
- Calibration scores can be misleading if prediction accuracy is poor, as uniform distributions minimize the score regardless of predictive quality.
- Locality (size consistency) is only satisfied in expectation for MCD due to its stochastic nature, unlike deterministic methods.

## Evidence (verbatim from paper)

> For OOD detection we use the area under the receiver operating characteristic curve (AUC-ROC). For each backbone, we use the respective hyperparameters from their papers, listed in Section A.2.

## Citation

```bibtex
@misc{wollschlaeger2023uncertainty,
  title={Uncertainty Estimation for Molecules: Desiderata and Methods},
  author={Wollschläger et al. (2023)},
  year={2023},
  note={arXiv:2306.14916}
}
```

- arXiv: 2306.14916