Matsciml Eval

---
name: matsciml-eval
description: Evaluates graph neural networks and equivariant point cloud networks on solid-state materials modeling tasks, including energy/force prediction, bandgap/fermi level regression, and crystal symmetry classification. Probes single-task, multi-task, and multi-dataset generalization capabilities. Use when the user wants to benchmark on OpenCatalyst (OC-20), Materials Project (MP), LiPS, OQMD, NOMAD, CMD, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05934
  bibtex_key: lee2023matsciml
  confidence: high
---

# matsciml-eval

> MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling — Lee et al. (2023) (arXiv:2309.05934, 2023)

## What this evaluates

Evaluates graph neural networks and equivariant point cloud networks on solid-state materials modeling tasks, including energy/force prediction, bandgap/fermi level regression, and crystal symmetry classification. Probes single-task, multi-task, and multi-dataset generalization capabilities.

## Datasets

- **OpenCatalyst (OC-20)** — total ?; splits: train (-1), val (-1), test (-1)
- **Materials Project (MP)** — total ?; splits: train (-1), val (-1), test (-1)
- **LiPS** — total ?; splits: train (-1), val (-1), test (-1)
- **OQMD** — total ?; splits: train (-1), val (-1), test (-1)
- **NOMAD** — total ?; splits: train (-1), val (-1), test (-1)
- **CMD** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error: (1/N) * Σ(y_true - y_pred)^2. Used for energy, bandgap, and fermi level prediction.
- `MAE` — range: other
  - Mean Absolute Error: (1/N) * Σ|y_true - y_pred|. Used for force prediction.
- `Accuracy` — range: [0, 1]
  - Correct predictions divided by total predictions. Used for stability and space group classification.

## Input / output format

**Input**: Crystal structures represented as graphs or point clouds, including atomic coordinates, species, and periodic boundary conditions.

**Output**: Predicted scalar values (energy, bandgap, fermi level), force vectors, or class labels (stability, space group).

## Scoring recipe

```python
def compute_metric(y_true, y_pred, task_type):
    if task_type in ['energy', 'bandgap', 'fermi']:
        return np.mean((y_true - y_pred) ** 2)  # MSE
    elif task_type == 'force':
        return np.mean(np.abs(y_true - y_pred))  # MAE
    elif task_type in ['stability', 'space_group']:
        return np.mean(y_true == y_pred)  # Accuracy
```

## Common pitfalls

- Comparing to OC20 leaderboard test splits vs. public validation splits leads to unfair comparisons.
- Space group classification suffers from severe class imbalance and hierarchical symmetry, making it inherently difficult for all models.
- Multi-data learning for IS2RE worsens performance because it predicts relaxed energy rather than single-frame energy, unlike other datasets.

## Evidence (verbatim from paper)

> The results from [Table 3] indicate that: Task Metric ... Energy Prediction S2EF MSE 0.826 ... Force Prediction S2EF MAE 0.957 ... Space ACC 29.8

## Citation

```bibtex
@misc{lee2023matsciml,
  title={MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2309.05934}
}
```

- arXiv: 2309.05934