Weather 5k Eval

---
name: weather-5k-eval
description: Evaluates the capability of data-driven time-series forecasting models to predict global meteorological variables over short to long horizons, and assesses their robustness in forecasting extreme weather events compared to numerical weather prediction baselines. Use when the user wants to benchmark on WEATHER-5K, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.14399
  bibtex_key: [REDACTED]
  confidence: high
---

# weather-5k-eval

> How far are today's time-series models from real-world weather forecasting applications? — Han et al. (2024) (arXiv:2406.14399, 2024)

## What this evaluates

Evaluates the capability of data-driven time-series forecasting models to predict global meteorological variables over short to long horizons, and assesses their robustness in forecasting extreme weather events compared to numerical weather prediction baselines.

## Datasets

- **WEATHER-5K** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/taohan10200/WEATHER-5K

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error; computes the average of absolute differences between predicted and observed values across all stations and time steps.
- `MSE` — range: other
  - Mean Square Error; computes the average of squared differences between predicted and observed values, amplifying outlier errors.
- `SEDI` — range: [0, 1]
  - Symmetric Extremal Dependence Index; calculates the ratio of correctly identified extreme events (both upper and lower quantile thresholds) to the total number of observed extreme events. SEDI ∈ [0,1].

## Input / output format

**Input**: Spatial-temporal time-series tensor X ∈ R^(N×T×V) representing N weather stations, T=48 historical time steps, and V meteorological variables.

**Output**: Predicted time-series tensor X̂ ∈ R^(N×τ×V) for forecast horizon τ ∈ {24, 72, 120, 168} steps.

## Scoring recipe

```python
def compute_metrics(pred, true):
    mae = np.mean(np.abs(pred - true))
    mse = np.mean((pred - true) ** 2)
    sedis = {}
    for var in V:
        for p in [0.90, 0.95, 0.98, 0.995, 0.10, 0.05, 0.02, 0.005]:
            q_lower = np.percentile(true[:, :, var], p * 100)
            q_upper = np.percentile(true[:, :, var], (1 - p) * 100)
            tp = np.sum((pred < q_lower) & (true < q_lower)) + np.sum((pred > q_upper) & (true > q_upper))
            fn = np.sum(true < q_lower) + np.sum(true > q_upper)
            sedis[(var, p)] = tp / fn if fn > 0 else 0.0
    return mae, mse, sedis
```

## Common pitfalls

- All baselines must use a fixed input length of 48 historical steps to ensure fair comparison, regardless of their original implementations.
- Results are reported from a single training run per model rather than averaged over multiple random seeds, relying on dataset stability.
- SEDI thresholds are computed station-specifically based on observed quantiles, not global dataset percentiles.

## Evidence (verbatim from paper)

> The WEATHER-5K dataset is divided into three subsets: training (with years 2014-2021), validation (with yea 2022), and test (with year 2023), which follows an 8:1:1 ratio. ... Mean Absolute Error (MAE) and Mean Square Error (MSE) are used to evaluate the overall performance of the GSWF. ... SEDI\in[0,1] quantifies the model’s ability to correctly identify extreme weather events.

## Citation

```bibtex
@misc{han2024weather5k,
  title={How far are today's time-series models from real-world weather forecasting applications?},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2406.14399}
}
```

- arXiv: 2406.14399