Ept15 Weather Bench Eval

---
name: ept15-weather-bench-eval
description: Evaluates the accuracy of AI weather forecasting models against established numerical models and ground-truth observations. It probes the model's ability to predict atmospheric variables (e.g., wind speed, solar radiation) at hourly resolution over 20-day lead times. Use when the user wants to benchmark on ERA5, IFS HRES IC, Weather Stations, or asks about evaluating this task. Reports Skill Score (SS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15076
  bibtex_key: molinaro2024ept15
  confidence: high
---

# ept15-weather-bench-eval

> EPT-1.5 Technical Report — Molinaro et al. (2024) (arXiv:2410.15076, 2024)

## What this evaluates

Evaluates the accuracy of AI weather forecasting models against established numerical models and ground-truth observations. It probes the model's ability to predict atmospheric variables (e.g., wind speed, solar radiation) at hourly resolution over 20-day lead times.

## Datasets

- **ERA5** — total ?; splits: test (-1)
- **IFS HRES IC** — total ?; splits: test (-1)
- **Weather Stations** — total ?; splits: test (-1)

## Metrics

- `Skill Score (SS)` **(primary)** — range: other (typically [-1, 1], unbounded above)
  - SS = 1 - (RMSE_model / RMSE_reference), where RMSE = sqrt(sum(w_i * (X_tilde_i - X_i)^2)). Compares model error against a reference forecast (IFS HRES). Higher values indicate better performance.

## Input / output format

**Input**: Gridded or point-wise initial weather conditions (ERA5 or IFS HRES IC) and reference forecast data.

**Output**: Predicted gridded or point-wise weather state variables (e.g., wind speed, solar radiation) at specified lead times.

## Scoring recipe

```python
def compute_skill_score(predictions, ground_truth, reference, weights):
    rmse_model = np.sqrt(np.sum(weights * (predictions - ground_truth)**2))
    rmse_ref = np.sqrt(np.sum(weights * (reference - ground_truth)**2))
    return 1.0 - (rmse_model / rmse_ref)
```

## Common pitfalls

- Using ERA5 as ground truth for IFS HRES forecasts introduces initial error (RMSE > 0 at t=0), unfairly penalizing the numerical model. The protocol mandates using consistent initial conditions or point observations.
- Weather station sensors are mounted at 2-10m height, not 100m, so direct comparison with model outputs at 100m requires caution or height adjustment.
- The weighting factor w_i in the RMSE formula is not explicitly defined in the text, which could lead to inconsistent aggregation across variables or grid cells.

## Evidence (verbatim from paper)

> Performance is evaluated based on the skill score (SS) defined as SS=1−RMSE_model/RMSE_reference, with RMSE denoting the Root Mean Square Error: RMSE=√∑_{i=1}^{n}w_{i}(˜X_{i}−X_{i})^{2}. Here, ˜X_{i} represents the predicted value, X_{i} is the weather state corresponding to the ground truth dataset, while the reference forecast is IFS HRES as outlined above.

## Citation

```bibtex
@misc{molinaro2024ept15,
  title={EPT-1.5 Technical Report},
  author={Molinaro et al. (2024)},
  year={2024},
  note={arXiv:2410.15076}
}
```

- arXiv: 2410.15076