Stormcast Forecast Eval

---
name: stormcast-forecast-eval
description: Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times. Use when the user wants to benchmark on ERA5, HRRR, MRMS, or asks about evaluating this task. Reports FSS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.10958
  bibtex_key: [REDACTED]
  confidence: high
---

# stormcast-forecast-eval

> Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling — Pathak et al. (2024) (arXiv:2408.10958, 2024)

## What this evaluates

Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times.

## Datasets

- **ERA5** — total ?; splits: (unstated)
- **HRRR** — total ?; splits: (unstated)
- **MRMS** — total ?; splits: (unstated)

## Metrics

- `FSS` **(primary)** — range: [0, 1]
  - Fractions Skill Score. Computed by binarizing forecast and reference fields at specified thresholds (20, 30, 40 dBZ), applying spatial pooling (15 or 45 km windows), and calculating fractional overlap: 1 - sum((F-R)^2) / sum(F^2 + R^2).
- `RMSE` — range: other
  - Root Mean Square Error computed relative to the HRRR analysis baseline across multiple atmospheric variables (radar reflectivity, wind, temperature, humidity).

## Input / output format

**Input**: 26 synoptic conditioning variables from ERA5 reanalysis, plus HRRR analysis fields at the initialization time step.

**Output**: 99 atmospheric state variables predicted at 1-hour intervals for 1-12 hour lead times, including composite radar reflectivity, horizontal wind components, temperature, and specific humidity at multiple vertical levels.

## Scoring recipe

```python
def compute_fss(forecast, reference, threshold, pool_size):
    f_bin = (forecast >= threshold).astype(float)
    r_bin = (reference >= threshold).astype(float)
    f_pool = spatial_average(f_bin, pool_size)
    r_pool = spatial_average(r_bin, pool_size)
    num = sum((f_pool - r_pool)**2)
    den = sum(f_pool**2 + r_pool**2)
    return 1 - (num / den)

def compute_rmse(forecast, reference):
    return sqrt(mean((forecast - reference)**2))
```

## Common pitfalls

- FSS values are highly sensitive to the chosen pooling window size and dBZ threshold; small changes drastically alter perceived skill.
- The 1-hour time resolution prevents direct temporal evolution analysis, requiring reliance on spatial/multivariate co-location checks instead.
- Ensemble Probability Matched Mean (PMM) masks individual member errors; evaluating single members vs. PMM yields different conclusions about skill.

## Evidence (verbatim from paper)

> The FSS of composite reflectivity forecasts are evaluated at thresholds of 20dBZ (light rain), 30dBZ (light-to-moderate rain) and 40dBZ (moderate rain). We compare the single member forecasts from StormCast (dotted lines) as well as the PMM of a five-member ensemble from the StormCast model (dashed lines) against the HRRR single member forecast baseline (solid lines).

## Citation

```bibtex
@misc{pathak2024stormcast,
  title={Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling},
  author={Pathak et al. (2024)},
  year={2024},
  note={arXiv:2408.10958}
}
```

- arXiv: 2408.10958