Ens 10 Eval

---
name: ens-10-eval
description: Probes the ability of deep learning and statistical models to correct biases in long-term ensemble weather forecasts. It evaluates how well models can post-process raw ensemble members to produce calibrated predictive distributions for surface and atmospheric variables. Use when the user wants to benchmark on ENS-10, or asks about evaluating this task. Reports CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.14786
  bibtex_key: ashkboos2022ens10
  confidence: high
---

# ens-10-eval

> ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts — Ashkboos et al. (2022) (arXiv:2206.14786, 2022)

## What this evaluates

Probes the ability of deep learning and statistical models to correct biases in long-term ensemble weather forecasts. It evaluates how well models can post-process raw ensemble members to produce calibrated predictive distributions for surface and atmospheric variables.

## Datasets

- **ENS-10** — total ?; splits: test (-1)

## Metrics

- `CRPS` **(primary)** — range: other (lower is better)
  - Continuous Ranked Probability Score. Assumes a Gaussian predictive distribution N(μ, σ²). CRPS(F, x) = σ[2ψ((x-μ)/σ) + (x-μ)/σ(2φ((x-μ)/σ)-1) - 1/√π], where ψ and φ are the standard Gaussian PDF and CDF, μ and σ are predicted mean and std, and x is the observation.
- `EECRPS` — range: other (lower is better)
  - Extreme Event-weighted CRPS. Computes CRPS for each grid point, weights it by the absolute value of the Extreme Forecast Index (EFI), and averages over the global test set.

## Input / output format

**Input**: Ensemble forecast members (5 or 10) for multiple weather variables at T=48h, optionally including spatial coordinates (lat/lon or xyz). Data is normalized per model (e.g., time-ensemble mean/std per grid point for MLP/U-Net, or global min/max for EMOS).

**Output**: Corrected mean (μ) and standard deviation (σ) of a Gaussian distribution for the target variable at each grid point. (Transformer outputs corrected ensemble members, from which μ and σ are computed offline).

## Scoring recipe

```python
def compute_crps(mu, sigma, x):
    z = (x - mu) / sigma
    return sigma * (2 * norm.pdf(z) + z * (2 * norm.cdf(z) - 1) - 1 / sqrt(pi))

def compute_ee_crps(crps_map, efi_map):
    return np.mean(crps_map * np.abs(efi_map))

# For each grid point in test set:
# 1. Get predicted mu, sigma and observed x
# 2. crps_val = compute_crps(mu, sigma, x)
# 3. For EECRPS, weight crps_val by |EFI_val|
# 4. Average CRPS or weighted CRPS over all grid points
```

## Common pitfalls

- CRPS is non-differentiable, requiring a Gaussian assumption to compute it during training.
- Input normalization strategies vary significantly across baselines (per-grid-point time-ensemble scaling vs. global min-max scaling), complicating direct comparison.
- EMOS performance can degrade with 10 ensemble members due to optimizer sensitivity, requiring careful tuning.

## Evidence (verbatim from paper)

> We now define several representative baselines for the prediction correction task on ENS-10, and evaluate them all with both CRPS and EECRPS using five and ten ensemble members as input. ... To overcome this issue, following previous work [15, 36, 39], we assume a Gaussian distribution on the target variable and learn the mean and standard deviation of this distribution. In this case, the CRPS from Eqn. (1) can be rewritten as $$ \operatorname {C R P S} \left(F ^ {i, j}, x\right) = \sigma \left[ 2 \psi \left(\frac {x - \mu}{\sigma}\right) + \frac {x - \mu}{\sigma} \left(2 \phi \left(\frac {x - \mu}{\sigma}\right) - 1\right) - \frac {1}{\sqrt {\pi}} \right], $$ ... We use the absolute value of EFI to weight the CRPS metric before averaging over the global data points in our test set.

## Citation

```bibtex
@misc{ashkboos2022ens10,
  title={ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts},
  author={Ashkboos et al. (2022)},
  year={2022},
  note={arXiv:2206.14786}
}
```

- arXiv: 2206.14786