Molecule3d Eval

---
name: molecule3d-eval
description: This benchmark evaluates the ability of graph neural networks to predict ground-state 3D molecular geometries directly from 2D molecular graphs, and subsequently assesses how well these predicted geometries improve downstream quantum property prediction (HOMO-LUMO gap). Use when the user wants to benchmark on Molecule3D, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.01717
  bibtex_key: xu2021molecule3d
  confidence: high
---

# molecule3d-eval

> Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs — Zhao Xu et al. (2021) (arXiv:2110.01717, 2021)

## What this evaluates

This benchmark evaluates the ability of graph neural networks to predict ground-state 3D molecular geometries directly from 2D molecular graphs, and subsequently assesses how well these predicted geometries improve downstream quantum property prediction (HOMO-LUMO gap).

## Datasets

- **Molecule3D** — total 4000000; splits: val (-1), test (-1); repo https://github.com/divelab/MoleculeX

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: $\frac{1}{m}\sum_{i=1}^{m}|\hat{y}_{i}-y_{i}|$, where $\hat{y}_{i}$ and $y_{i}$ are predicted and ground-truth values (pairwise distances, coordinates, or HOMO-LUMO gaps). Lower values indicate better performance.
- `RMSE` — range: other
  - Root Mean Squared Error: $\sqrt{\frac{1}{m}\sum_{i=1}^{m}(\hat{y}_{i}-y_{i})^2}$. Sensitive to large errors in distance or coordinate prediction.
- `Validity` — range: percent
  - Percentage of predicted geometries that form a valid Euclidean Distance Matrix (EDM) and can be transformed into 3D coordinates.
- `Validity3D` — range: percent
  - Percentage of predicted geometries that are valid in 3D space (dimension ≤ 3) and satisfy triangle inequalities.

## Input / output format

**Input**: Molecular graph with 9-dimensional node features (atomic number, chirality, hybridization) and 3-dimensional edge features (bond type, stereochemistry, conjugation).

**Output**: Predicted pairwise atomic distances (via element-wise max and linear transformation of node representations) or predicted 3D atomic coordinates (3D vectors). For property prediction: scalar HOMO-LUMO gap value.

## Scoring recipe

```python
def compute_mae(preds, gold):
    return np.mean(np.abs(preds - gold))

def compute_rmse(preds, gold):
    return np.sqrt(np.mean((preds - gold) ** 2))

def compute_validity(preds):
    # Check if predicted EDM is valid and transformable to 3D
    return np.mean(is_valid_3d_geometry(preds)) * 100
```

## Common pitfalls

- Scaffold split is significantly harder than random split due to unseen molecular scaffolds, causing substantial performance drops.
- Predicting pairwise distances yields lower MAE/RMSE but extremely low Validity/Validity3D, while direct coordinate prediction guarantees 100% validity but higher distance errors.
- RDKit ETKDG baseline fails to generate geometries for a non-trivial number of molecules (1,311–4,434 depending on split), which must be accounted for when comparing success rates.

## Evidence (verbatim from paper)

> We evaluate the prediction performance by the mean absolute error (MAE) between the predicted properties and the ground-truth properties. Given a dataset of $m$ molecules whose HOMO-LUMO gaps are ${\hat{y}_{i}}_{i\=1}^{m}$, and the predicted HOMO-LUMO gaps are ${y_{i}}_{i\=1}^{m}$, where $y_{i},\hat{y}_{i}\in\mathbb{R}$, the MAE is defined as: $\mbox{MAE}\left({\hat{y}_{i}}_{i\=1}^{m},{y_{i}}_{i\=1}^{m}\right)\=\frac{1}{m}\sum_{i\=1}^{m}|\hat{y}_{i}-y_{i}|.$

## Citation

```bibtex
@misc{xu2021molecule3d,
  title={Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs},
  author={Zhao Xu et al. (2021)},
  year={2021},
  note={arXiv:2110.01717}
}
```

- arXiv: 2110.01717