M2hub Eval

---
name: m2hub-eval
description: Evaluates graph neural networks on predicting material properties from 3D crystal and molecular structures. It probes model performance across diverse material types, physical properties, and realistic data partitioning strategies. Use when the user wants to benchmark on OMDB, QMOF, MP, ISMETAL, EDOS, PDOS, DF2D, Perovskites, EFORM, Phonons, Dielectric, LOG_GVRH, LOG_KVRH, OC20, QM9, or asks about evaluating this task. Reports MAE / Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.05378
  bibtex_key: du2023m2hub
  confidence: high
---

# m2hub-eval

> M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery — Du et al. (2023) (arXiv:2307.05378, 2023)

## What this evaluates

Evaluates graph neural networks on predicting material properties from 3D crystal and molecular structures. It probes model performance across diverse material types, physical properties, and realistic data partitioning strategies.

## Datasets

- **OMDB** — total ?; splits: random (-1), composition (-1), system (-1)
- **QMOF** — total ?; splits: random (-1), composition (-1), system (-1)
- **MP** — total ?; splits: random (-1), composition (-1), system (-1)
- **ISMETAL** — total ?; splits: random (-1), composition (-1), system (-1)
- **EDOS** — total ?; splits: random (-1), composition (-1), system (-1)
- **PDOS** — total ?; splits: random (-1), composition (-1), system (-1)
- **DF2D** — total ?; splits: random (-1), composition (-1), system (-1)
- **Perovskites** — total ?; splits: random (-1), composition (-1), system (-1)
- **EFORM** — total ?; splits: random (-1), composition (-1), system (-1)
- **Phonons** — total ?; splits: random (-1), composition (-1), system (-1)
- **Dielectric** — total ?; splits: random (-1), composition (-1), system (-1)
- **LOG_GVRH** — total ?; splits: random (-1), composition (-1), system (-1)
- **LOG_KVRH** — total ?; splits: random (-1), composition (-1), system (-1)
- **OC20** — total ?; splits: random (-1), composition (-1), system (-1)
- **QM9** — total ?; splits: random (-1), composition (-1), system (-1)

## Metrics

- `MAE / Accuracy` **(primary)** — range: other
  - Mean Absolute Error (MAE) for regression tasks (e.g., bandgap, formation energy, density of states); classification accuracy for ISMETAL. Lower MAE and higher accuracy indicate better performance.

## Input / output format

**Input**: 3D atomic structures represented as graphs with node features (atom types) and edge features (pairwise distances, lattice vectors).

**Output**: Continuous property value for regression tasks; binary class label for classification tasks.

## Scoring recipe

```python
def compute_metric(predictions, targets, task_type):
    if task_type == 'classification':
        return accuracy_score(targets, predictions)
    else:
        return mean_absolute_error(targets, predictions)
```

## Common pitfalls

- System and composition splits prevent data leakage by separating entire systems or chemical compositions, making evaluation significantly harder than random splits.
- Some tasks have extremely small test sets after splitting, leading to missing results that should be noted.
- Efficiency varies drastically across models; runtime should be reported alongside accuracy for fair comparison.

## Evidence (verbatim from paper)

> We test all the methods on a list of 13 representative tasks from our benchmarks with three data splits (random, composition and system). ... Benchmark on materials property prediction tasks (different colors denote distinct property types: purple (electrical), yellow (stability), green (thermal), red (optical), blue (mechanical)... Table 3 reports the MAE / Accuracy for each task.)

## Citation

```bibtex
@misc{du2023m2hub,
  title={M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery},
  author={Du et al. (2023)},
  year={2023},
  note={arXiv:2307.05378}
}
```

- arXiv: 2307.05378