Weatherbench2 Eval

---
name: weatherbench2-eval
description: Evaluates the capability of generative weather forecasting models to predict global atmospheric and surface conditions from mid-range to sub-seasonal horizons (up to 30 days). It probes deterministic accuracy and probabilistic ensemble calibration against established meteorological baselines and climatology. Use when the user wants to benchmark on WeatherBench-2, or asks about evaluating this task. Reports Latitude-weighted RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.24428
  bibtex_key: [REDACTED]
  confidence: high
---

# weatherbench2-eval

> Marchuk: Efficient Global Weather Forecasting from Mid-Range to Sub-Seasonal Scales via Flow Matching — Kuzhamuratov et al. (2026) (arXiv:2603.24428, 2026)

## What this evaluates

Evaluates the capability of generative weather forecasting models to predict global atmospheric and surface conditions from mid-range to sub-seasonal horizons (up to 30 days). It probes deterministic accuracy and probabilistic ensemble calibration against established meteorological baselines and climatology.

## Datasets

- **WeatherBench-2** — total ?; splits: test (100)

## Metrics

- `Latitude-weighted RMSE` **(primary)** — range: other (physical units, e.g., m/s, K, m^2/s^2, hPa)
  - Root Mean Squared Error weighted by latitude to account for grid cell area distortion in spherical coordinates. Computed as the square root of the mean squared difference between predictions and observations, averaged over time and latitude.
- `Anomaly Correlation Coefficient (ACC)` — range: [-1, 1]
  - Pearson correlation coefficient computed between forecast anomalies and observation anomalies relative to the climatological mean. Ranges from -1 to 1, where 1 indicates perfect anomaly prediction.
- `Continuous Ranked Probability Score (CRPS)` — range: other (physical units)
  - Integral over the forecast cumulative distribution function of the squared difference between the CDF and a step function at the observed value. Measures the accuracy of probabilistic forecasts.
- `CRPS Skill` — range: [0, 1]
  - Relative improvement of the model's CRPS over a climatology baseline, calculated as 1 - (CRPS_model / CRPS_climatology).
- `CRPS Spread` — range: other (physical units)
  - Mean pairwise distance between ensemble members, quantifying the internal dispersion or uncertainty of the probabilistic forecast.

## Input / output format

**Input**: Multi-variable atmospheric and surface weather fields (e.g., wind, temperature, geopotential, humidity, pressure) over a temporal context window (1 day for Marchuk, 6 hours for baselines), provided as latent representations to a DiT architecture.

**Output**: Predicted multi-variable atmospheric and surface fields for forecast horizons up to 30 days, generated as deterministic point forecasts or probabilistic ensembles (e.g., 250 members).

## Scoring recipe

```python
def compute_metrics(pred, obs, lat_weights=None, ensemble=None):
    # Deterministic latitude-weighted RMSE
    if lat_weights is not None:
        rmse = np.sqrt(np.mean(lat_weights * (pred - obs)**2))
    else:
        rmse = np.sqrt(np.mean((pred - obs)**2))
    # ACC
    pred_anom = pred - np.mean(pred, axis=0)
    obs_anom = obs - np.mean(obs, axis=0)
    acc = np.cov(pred_anom.flatten(), obs_anom.flatten())[0,1] / np.sqrt(np.var(pred_anom)*np.var(obs_anom))
    # CRPS (ensemble)
    if ensemble is not None:
        crps = np.mean(np.abs(ensemble - obs))
    else:
        crps = None
    return rmse, acc, crps
```

## Common pitfalls

- Forgetting to apply latitude weighting when computing RMSE, which is explicitly required by the WeatherBench-2 framework to account for spherical grid distortion.
- Comparing models trained with different context windows (1 day vs 6 hours) without noting that longer context improves trajectory agreement but may artificially reduce ensemble spread.
- Using initialization dates outside the specified test years (2018 and 2021) or not sampling exactly 100 dates uniformly as mandated by the protocol.

## Evidence (verbatim from paper)

> Model performance is evaluated using the WeatherBench-2 evaluation framework. We report both deterministic and probabilistic forecast metrics. Deterministic metrics include: Latitude-weighted Root Mean Squared Error (RMSE), Anomaly Correlation Coefficient (ACC). Probabilistic metrics include: Continuous Ranked Probability Score (CRPS), CRPS Skill, CRPS Spread. Metrics are averaged over latitude and time and evaluated on standard meteorological variables such as Z500 and T850.

## Citation

```bibtex
@misc{kuzhamuratov2026marchuk,
  title={Marchuk: Efficient Global Weather Forecasting from Mid-Range to Sub-Seasonal Scales via Flow Matching},
  author={Kuzhamuratov et al. (2026)},
  year={2026},
  note={arXiv:2603.24428}
}
```

- arXiv: 2603.24428