S2s Ai Challenge Eval

---
name: s2s-ai-challenge-eval
description: Evaluates the skill of deep learning post-processing models for global sub-seasonal temperature and precipitation forecasts against climatological baselines and ECMWF recalibrated forecasts. It probes the ability of spatial CNN architectures to correct systematic errors and produce well-calibrated probabilistic tercile predictions over a 2–4 week horizon. Use when the user wants to benchmark on S2S AI Challenge test set (2020), or asks about evaluating this task. Reports RPSS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.15956
  bibtex_key: horat2023deeplearning
  confidence: high
---

# s2s-ai-challenge-eval

> Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales — Horat et al. (2023) (arXiv:2306.15956, 2023)

## What this evaluates

Evaluates the skill of deep learning post-processing models for global sub-seasonal temperature and precipitation forecasts against climatological baselines and ECMWF recalibrated forecasts. It probes the ability of spatial CNN architectures to correct systematic errors and produce well-calibrated probabilistic tercile predictions over a 2–4 week horizon.

## Datasets

- **S2S AI Challenge test set (2020)** — total 53; splits: test (53); repo https://github.com/HoratN/pp-s2s

## Metrics

- `RPSS` **(primary)** — range: [-1, 1]
  - RPSS = 1 − (RPS_model / RPS_climatology), where RPS is the Ranked Probability Score averaged over the 53 weekly forecasts. Positively oriented; higher values indicate better skill relative to climatology.

## Input / output format

**Input**: Global spatial forecast fields (ensemble mean quantities) from ECMWF for temperature and precipitation.

**Output**: Probabilistic tercile forecasts (probabilities for above, near, and below normal categories) for each grid cell.

## Scoring recipe

```python
def compute_rpss(predictions, observations, climatology_probs):
    # predictions, observations, climatology_probs: shape (num_forecasts, num_grid_cells, 3_terciles)
    rps_model = mean((predictions - observations) ** 2, axis=0)
    rps_clim = mean((climatology_probs - observations) ** 2, axis=0)
    rpss_grid = 1 - (rps_model / rps_clim)
    # Weighted average over land cells, excluding Antarctica and dry precip regions
    rpss_global = weighted_mean(rpss_grid, weights=land_area_fraction)
    return rpss_global
```

## Common pitfalls

- Only land grid cells are considered; sea cells are excluded due to lack of observations.
- Antarctica is explicitly excluded from the aggregated RPSS calculation.
- For precipitation, very dry regions (lower tercile edge < 0.01 mm/m^2) are omitted from evaluation.
- The metric is averaged over exactly 53 weekly forecasts issued on Thursdays in 2020.

## Evidence (verbatim from paper)

> Following the setup of the S2S AI Challenge, we evaluate the proposed post-processing methods based on the ranked probability skill score (RPSS) for the test set (year 2020), using the climatological forecast as a reference. The RPSS is based on a strictly proper scoring rule and introduced in detail in Appendix [A](#A1 "Appendix A Evaluation metrics ‣ Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales"). It is positively oriented, i.e., larger values indicate better forecasts. Following the challenge configuration, only land grid cells are considered for the computation of the spatially aggregated RPSS scores since no observations are available for sea grid cells.
Antarctica is not included in the aggregated RPSS. For precipitation, also very dry regions (lower tercile edge smaller than $0.01\text{\,}\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{m}^{2}$ are not taken into account.

## Citation

```bibtex
@misc{horat2023deeplearning,
  title={Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales},
  author={Horat et al. (2023)},
  year={2023},
  note={arXiv:2306.15956}
}
```

- arXiv: 2306.15956