Modnet Eval Protocol

---
name: modnet-eval-protocol
description: Evaluates the predictive accuracy and uncertainty quantification of a descriptor-based neural network on materials property prediction tasks, with a focus on low-data regimes and bias-imbalance. Use when the user wants to benchmark on Naccarato et al. refractive index, Petretto et al. vibrational thermodynamics, or asks about evaluating this task. Reports mean absolute error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.02263
  bibtex_key: debreuck2021modnet
  confidence: medium
---

# modnet-eval-protocol

> Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet — De Breuck et al. (2021) (arXiv:2102.02263, 2021)

## What this evaluates

Evaluates the predictive accuracy and uncertainty quantification of a descriptor-based neural network on materials property prediction tasks, with a focus on low-data regimes and bias-imbalance.

## Datasets

- **Naccarato et al. refractive index** — total 4040; splits: train (-1), test (-1)
- **Petretto et al. vibrational thermodynamics** — total 1265; splits: train (-1), test (-1)

## Metrics

- `mean absolute error` **(primary)** — range: other
  - Average of absolute differences between predicted and true property values across the test set.
- `standard deviation` — range: other
  - Standard deviation of predictions across the 125-model ensemble, used as an epistemic uncertainty measure.

## Input / output format

**Input**: Material composition or crystal structure processed into chemical, physical, and geometrical descriptors using Matminer featurizers.

**Output**: Scalar prediction (ensemble mean) and uncertainty estimate (ensemble standard deviation). Predictions falling outside the training target range padded by 25% are remapped to a uniform distribution within that range.

## Scoring recipe

```python
predictions = [model_i(x) for model_i in ensemble]
mean_pred = np.mean(predictions)
uncertainty = np.std(predictions)
r = y_max - y_min
if mean_pred < y_min - 0.25*r or mean_pred > y_max + 0.25*r:
    mean_pred = np.random.uniform(y_min, y_max)
mae = np.mean(np.abs(mean_pred - y_true))
```

## Common pitfalls

- Hyperparameters must be optimized via nested cross-validation (internal validation folds), never directly on the test set to avoid overfitting.
- Predictions must be checked against the training target range (padded by 25%); out-of-range values are replaced with random draws to ensure physical plausibility.

## Evidence (verbatim from paper)

> MODNet has been trained and tested on both single and multi-property tasks with excellent performance compared to contemporary methods for datasets containing 10,000 samples or fewer. For instance, on the refractive index dataset of Naccaratoet al. (4040 samples), MODNet achieves a mean absolute error of 0.051... The specific hyperparameter optimization process used in this work is described in [B]... automated hyperparameter selection ensures a correct assessment of the generalisation error and minimises overfitting. That is, hyperparameters should never be chosen to minimise a test error, but rather chosen internally with a validation procedure.

## Citation

```bibtex
@misc{debreuck2021modnet,
  title={Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet},
  author={De Breuck et al. (2021)},
  year={2021},
  note={arXiv:2102.02263}
}
```

- arXiv: 2102.02263