Weatherbench Probability Eval

---
name: weatherbench-probability-eval
description: Evaluates the accuracy and reliability of probabilistic medium-range weather forecasting models against operational ensemble baselines. It probes how well deep learning methods capture uncertainty, calibration, and sharpness for key atmospheric variables. Use when the user wants to benchmark on WeatherBench Probability, or asks about evaluating this task. Reports CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.00865
  bibtex_key: garg2022weatherbenchprobability
  confidence: high
---

# weatherbench-probability-eval

> WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models — Garg et al. (2022) (arXiv:2205.00865, 2022)

## What this evaluates

Evaluates the accuracy and reliability of probabilistic medium-range weather forecasting models against operational ensemble baselines. It probes how well deep learning methods capture uncertainty, calibration, and sharpness for key atmospheric variables.

## Datasets

- **WeatherBench Probability** — total ?; splits: test (-1)

## Metrics

- `RMSE of ensemble mean` — range: other
  - Root mean squared error between the ensemble mean forecast and observations. Lower values indicate better point accuracy.
- `Spread-skill ratio` — range: other
  - Ratio of the ensemble spread (standard deviation of members) to the RMSE of the ensemble mean. A value near 1 indicates optimal dispersion; <1 means underdispersive, >1 means overdispersive.
- `CRPS` **(primary)** — range: other
  - Continuous Ranked Probability Score, which evaluates both calibration and sharpness of probabilistic forecasts against observations. Lower values indicate better probabilistic skill.

## Input / output format

**Input**: Atmospheric state variables (e.g., 500hPa geopotential height Z500, 850hPa temperature T850, 2m temperature T2M, total precipitation TP) for medium-range forecasting.

**Output**: Probabilistic forecasts: ensemble members, parametric distribution parameters, or categorical class probabilities.

## Scoring recipe

```python
def compute_metrics(pred_ensemble, pred_mean, obs):
    rmse = np.sqrt(np.mean((pred_mean - obs) ** 2))
    spread = np.std(pred_ensemble, axis=0)
    spread_skill = spread / rmse
    # CRPS for ensemble forecasts
    n = len(pred_ensemble)
    crps = (1/n) * np.sum(np.abs(pred_ensemble - obs)) - (1/(2*n**2)) * np.sum(np.abs(pred_ensemble[:, None] - pred_ensemble))
    return rmse, spread_skill, crps
```

## Common pitfalls

- RMSE is not a suitable metric for highly intermittent and skewed fields like precipitation.
- TIGGE baseline results are not post-processed, which likely underestimates their calibration and spread-skill ratio.
- MC dropout severely underestimates uncertainty (underdispersion) unless dropout rate is carefully tuned.

## Evidence (verbatim from paper)

> The ensemble mean RMSE and the CRPS is lowest for a dropout rate of 0.1. The spread-skill ratio shows that the dropout ensemble is severely underdispersive with the spread being less than half of what it should be.

## Citation

```bibtex
@misc{garg2022weatherbenchprobability,
  title={WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models},
  author={Garg et al. (2022)},
  year={2022},
  note={arXiv:2205.00865}
}
```

- arXiv: 2205.00865