The Well Eval

---
name: the-well-eval
description: Evaluates deep learning surrogate models on autoregressive time-series prediction across 16 diverse physics simulations. It probes the model's ability to forecast future spatiotemporal states from short historical snapshots and maintain stability over longer rollout horizons. Use when the user wants to benchmark on The Well, or asks about evaluating this task. Reports VRMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.00568
  bibtex_key: ohana2024thewell
  confidence: high
---

# the-well-eval

> The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning — Ohana et al. (2024) (arXiv:2412.00568, 2024)

## What this evaluates

Evaluates deep learning surrogate models on autoregressive time-series prediction across 16 diverse physics simulations. It probes the model's ability to forecast future spatiotemporal states from short historical snapshots and maintain stability over longer rollout horizons.

## Datasets

- **The Well** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/PolymathicAI/the_well

## Metrics

- `VRMSE` **(primary)** — range: other
  - Variance Scaled Root Mean Squared Error. Scaled such that predicting the mean value of the target field yields a score of 1. Computed as RMSE divided by the RMSE of a mean-field baseline, averaged over all physical fields.

## Input / output format

**Input**: A sequence of 4 temporally coarsened, uniformly sampled grid snapshots representing the short history of a physical simulation.

**Output**: The next single grid snapshot of the simulation.

## Scoring recipe

```python
def compute_vrmse(pred, true):
    rmse = np.sqrt(np.mean((pred - true) ** 2))
    mean_pred = np.full_like(true, np.mean(true))
    norm = np.sqrt(np.mean((mean_pred - true) ** 2))
    return rmse / norm
```

## Common pitfalls

- One-step evaluation uses sliding windows sampled from ground truth, while multi-step rollouts always start from the beginning of the simulation, causing performance gaps that reflect physics behavior rather than just model capability.
- NRMSE (using 2-norm normalization) down-weights errors for non-negative fields like pressure or density that have little variation; VRMSE is preferred for these cases.
- Boundary conditions are handled naively according to model defaults, which can unfairly penalize or favor certain architectures (e.g., U-net vs FNO) without a clear trend.

## Evidence (verbatim from paper)

> Table [2] reports the one-step Variance Scaled Root Mean Squared Error (VRMSE) – defined in Section [E.3] – averaged over all physical fields. We report evaluation on the test set of each model with hyperparameters performing best on the validation set.

## Citation

```bibtex
@misc{ohana2024thewell,
  title={The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning},
  author={Ohana et al. (2024)},
  year={2024},
  note={arXiv:2412.00568}
}
```

- arXiv: 2412.00568