Boom Ood Eval

---
name: boom-ood-eval
description: Evaluates the out-of-distribution (OOD) generalization of machine learning models for molecular property prediction. It probes whether models trained on in-distribution (ID) molecules can accurately extrapolate to novel chemical spaces, highlighting the disconnect between ID accuracy and OOD robustness. Use when the user wants to benchmark on QM9, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.01912
  bibtex_key: antoniuk2025boom
  confidence: high
---

# boom-ood-eval

> BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models — Antoniuk et al. (2025) (arXiv:2505.01912, 2025)

## What this evaluates

Evaluates the out-of-distribution (OOD) generalization of machine learning models for molecular property prediction. It probes whether models trained on in-distribution (ID) molecules can accurately extrapolate to novel chemical spaces, highlighting the disconnect between ID accuracy and OOD robustness.

## Datasets

- **QM9** — total ?; splits: train (-1), test (-1), ood_test (10000)

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error: sqrt(mean((y_pred - y_true)^2)). Used as the primary metric for both ID and OOD performance across all tasks.
- `R²` — range: [0, 1]
  - Coefficient of Determination: 1 - (sum((y_true - y_pred)^2) / sum((y_true - mean(y_true))^2)). Standard R² is reported for ID splits, while binned R² is used for OOD splits.

## Input / output format

**Input**: Molecular structures provided as 3D coordinates, graph representations, or SMILES strings.

**Output**: Continuous numerical values corresponding to specific molecular properties (e.g., heat of formation, density, HOMO/LUMO energies, dipole moment, heat capacity).

## Scoring recipe

```python
import numpy as np
def compute_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))
def compute_r2(y_true, y_pred):
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    return 1 - (ss_res / ss_tot)
```

## Common pitfalls

- ID performance does not reliably predict OOD performance; models can overfit to ID space while failing to extrapolate.
- Autoregressive models often fail on OOD splits due to numerical token generation errors (e.g., predicting '00913' for '0.913'), producing characteristic S-shaped parity plots.
- Standard masked language modeling pretraining improves ID accuracy but can significantly degrade OOD performance (e.g., binned R² drops by 39-53%).

## Evidence (verbatim from paper)

> The Geoformer achieves the best overall ID performance, achieving the lowest ID RMSE on 3 out of 10 tasks. For OOD prediction, GotenNet achieves top performance on 7 out of 10 tasks, and MACE achieves top performance on 2 out of 10 tasks.

## Citation

```bibtex
@misc{antoniuk2025boom,
  title={BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models},
  author={Antoniuk et al. (2025)},
  year={2025},
  note={arXiv:2505.01912}
}
```

- arXiv: 2505.01912