Duq Weather Forecasting Eval

---
name: duq-weather-forecasting-eval
description: Evaluates a deep learning model's capability to perform spatio-temporal weather forecasting and quantify predictive uncertainty. It tests the model's ability to fuse historical observations with numerical weather prediction (NWP) data to generate accurate point forecasts and reliable 90% prediction intervals over a 37-hour horizon. Use when the user wants to benchmark on Beijing weather dataset, or asks about evaluating this task. Reports SS_avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1812.09467
  bibtex_key: wang2018deepuncertainty
  confidence: high
---

# duq-weather-forecasting-eval

> Deep Uncertainty Quantification: A Machine Learning Approach for Weather Forecasting — Bin Wang et al. (arXiv:1812.09467, 2018)

## What this evaluates

Evaluates a deep learning model's capability to perform spatio-temporal weather forecasting and quantify predictive uncertainty. It tests the model's ability to fuse historical observations with numerical weather prediction (NWP) data to generate accurate point forecasts and reliable 90% prediction intervals over a 37-hour horizon.

## Datasets

- **Beijing weather dataset** — total ?; splits: train (11480), val (870), test (90)

## Metrics

- `SS_avg` **(primary)** — range: [-1, 1]
  - Average Skill Score across 9 test days, calculated as 1 - (RMSE_model / RMSE_baseline). Measures relative improvement over a baseline forecast.
- `RMSE_avg` — range: other
  - Average Root Mean Square Error across 9 test days. Computed as the square root of the mean squared difference between predicted and observed values.
- `PICP_avg` — range: [0, 1]
  - Average Prediction Interval Coverage Probability across 9 test days. Represents the fraction of ground truth values that fall within the predicted 90% interval.

## Input / output format

**Input**: Encoder inputs: historical observations (28 hours × 10 stations × 9 variables), NWP forecasts, and time/station IDs. Decoder inputs: concatenated NWP forecasts and IDs.

**Output**: Predicted values for 3 target variables (t2m, rh2m, w10m) over 37 forecast hours, along with corresponding 90% prediction intervals.

## Scoring recipe

```python
def compute_metrics(preds, targets, intervals, nwp_preds):
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    rmse_nwp = np.sqrt(np.mean((nwp_preds - targets) ** 2))
    ss = 1 - (rmse / rmse_nwp)
    picp = np.mean((targets >= intervals[:, 0]) & (targets <= intervals[:, 1]))
    return ss, rmse, picp
```

## Common pitfalls

- SS and RMSE rankings can diverge because they are not linearly related; a model optimizing one may not optimize the other.
- The test set is extremely small (9 days), so statistical significance relies on a one-tail paired T-test with a relaxed significance level of 0.25.
- Prediction interval widths do not monotonically increase over the forecast horizon due to diurnal weather patterns (narrower at night, wider during daytime fluctuations).

## Evidence (verbatim from paper)

> We also evaluated all methods by $RMSE_{avg}$ as shown in Table [2]. Since $RMSE_{avg}$ and $SS_{avg}$ do not have a fully linear relationship, the counterpart assessment does not reach the optimum at the same time while DUQEsb10 still achieves the best $RMSE_{avg}$.

## Citation

```bibtex
@misc{wang2018deepuncertainty,
  title={Deep Uncertainty Quantification: A Machine Learning Approach for Weather Forecasting},
  author={Bin Wang et al.},
  year={2018},
  note={arXiv:1812.09467}
}
```

- arXiv: 1812.09467