Subseasonal Climate Usa Eval

---
name: subseasonal-climate-usa-eval
description: Evaluates the ability of machine learning and dynamical models to forecast subseasonal temperature and precipitation over the contiguous United States at 3–4 and 5–6 week lead times. It benchmarks predictive accuracy against operational baselines and tests robustness to high noise and spatial heterogeneity in climate data. Use when the user wants to benchmark on SubseasonalClimateUSA, or asks about evaluating this task. Reports % IMPROVEMENT OVER MEAN DEB. CFSV2 RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.10399
  bibtex_key: mouatadid2021subseasonalclimateusa
  confidence: high
---

# subseasonal-climate-usa-eval

> SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking — Mouatadid et al. (2021) (arXiv:2109.10399, 2021)

## What this evaluates

Evaluates the ability of machine learning and dynamical models to forecast subseasonal temperature and precipitation over the contiguous United States at 3–4 and 5–6 week lead times. It benchmarks predictive accuracy against operational baselines and tests robustness to high noise and spatial heterogeneity in climate data.

## Datasets

- **SubseasonalClimateUSA** — total ?; splits: test (-1); repo https://github.com/microsoft/subseasonal_data

## Metrics

- `% IMPROVEMENT OVER MEAN DEB. CFSV2 RMSE` **(primary)** — range: percent
  - Calculated as (RMSE_baseline - RMSE_model) / RMSE_baseline * 100. Measures the relative reduction in root mean squared error compared to the debiased CFSv2 baseline.
- `AVERAGE % SKILL` — range: percent
  - Identical to RMSE improvement in this protocol: (RMSE_baseline - RMSE_model) / RMSE_baseline * 100. Used interchangeably with RMSE improvement to report model skill relative to the baseline.

## Input / output format

**Input**: Spatiotemporal ground truth observations and operational model forecasts (e.g., CFSv2, ECMWF) for the contiguous U.S., typically on a 1.5°×1.5° or 1°×1° grid.

**Output**: Predicted temperature and precipitation values for lead times of 3–4 weeks and 5–6 weeks, issued weekly.

## Scoring recipe

```python
def compute_metrics(preds, gold, baseline):
    rmse_pred = np.sqrt(np.mean((preds - gold)**2))
    rmse_base = np.sqrt(np.mean((baseline - gold)**2))
    rmse_improvement = ((rmse_base - rmse_pred) / rmse_base) * 100
    skill = ((rmse_base - rmse_pred) / rmse_base) * 100
    return rmse_improvement, skill
```

## Common pitfalls

- Dynamical model forecasts must be debiased before fair comparison, otherwise RMSE/skill scores are artificially penalized.
- Complex deep learning models often overfit due to high noise levels in subseasonal data, making simpler hybrid or baseline models more robust.
- Performance is highly heterogeneous across space and time; averaging over the entire contiguous U.S. can mask regional strengths or weaknesses.

## Evidence (verbatim from paper)

> We generate forecasts for each Wednesday in the years 2011-2020 and, for each reported period, we assess both mean RMSE relative to a baseline model and average percentage skill.

## Citation

```bibtex
@misc{mouatadid2021subseasonalclimateusa,
  title={SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking},
  author={Mouatadid et al. (2021)},
  year={2021},
  note={arXiv:2109.10399}
}
```

- arXiv: 2109.10399