Era5 Weatherbench Forecasting Eval

---
name: era5-weatherbench-forecasting-eval
description: Spatio-temporal forecasting of atmospheric temperature using deep learning models. It probes the ability to capture long-range spatial-temporal dependencies and predict future weather states from historical multi-feature sequences. Use when the user wants to benchmark on ERA5 Turkey, WeatherBench, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.00696
  bibtex_key: tekin2021numericalweather
  confidence: high
---

# era5-weatherbench-forecasting-eval

> Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms — Tekin et al. (2021) (arXiv:2102.00696, 2021)

## What this evaluates

Spatio-temporal forecasting of atmospheric temperature using deep learning models. It probes the ability to capture long-range spatial-temporal dependencies and predict future weather states from historical multi-feature sequences.

## Datasets

- **ERA5 Turkey** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/sftekin/ieee_weather
- **WeatherBench** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error: sqrt(mean((y_pred - y_true)^2)). Measures the standard deviation of prediction errors across all grid cells and time steps.
- `MAE` — range: other
  - Mean Absolute Error: mean(|y_pred - y_true|). Measures the average magnitude of errors without considering direction.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: mean(|(y_true - y_pred) / y_true|) * 100. Expresses accuracy as a percentage relative to ground truth values.
- `ACC` — range: [-1, 1]
  - Anomaly Correlation Coefficient: 1 - sum((y_pred - y_true)^2) / sum((y_true - y_mean)^2). Measures the correlation of forecast anomalies relative to the climatological mean.

## Input / output format

**Input**: Spatio-temporal sequences of weather features (e.g., temperature, wind, humidity) on a 2D grid. Input length is typically 10 time steps (3-hour frequency) or adapted per dataset.

**Output**: Predicted temperature values for the next N time steps on the same 2D grid. Output length varies by method: 5 steps (sequential ERA5), 72 steps (sequential WeatherBench), 6 steps (iterative WeatherBench), or 3 steps (direct forecast).

## Scoring recipe

```python
import numpy as np
def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_pred - y_true)**2))
    mae = np.mean(np.abs(y_pred - y_true))
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    y_mean = np.mean(y_true)
    acc = 1 - np.sum((y_pred - y_true)**2) / np.sum((y_true - y_mean)**2)
    return {'RMSE': rmse, 'MAE': mae, 'MAPE': mape, 'ACC': acc}
```

## Common pitfalls

- Confusion between sequential, iterative, and direct forecasting protocols leads to mismatched output lengths and invalid comparisons.
- Table III contains a copy-paste error where the Weather Model's RMSE/MAE/MAPE values are identical to Table II, likely a typo in the original publication.
- Metrics are computed on raw temperature values, but inputs are min-max normalized per batch, requiring inverse transformation before metric calculation.

## Evidence (verbatim from paper)

> We split the dataset into the train, validation and test sets, with 0.8, 0.1 and 0.1 split ratios, respectively. ... The numerical results of the experiment are illustrated in Table II. We illustrate the performance of the models on the high-resolution dataset. ... TABLE II: We illustrate the performance of the models on the high-resolution dataset. The best results for each metric are given in bold. | Model Name | RMSE | MAE | MAPE |

## Citation

```bibtex
@misc{tekin2021numericalweather,
  title={Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms},
  author={Tekin et al. (2021)},
  year={2021},
  note={arXiv:2102.00696}
}
```

- arXiv: 2102.00696