Climate Set Ood Eval

---
name: climate-set-ood-eval
description: Evaluates the out-of-distribution robustness of machine learning climate emulation models under time-domain shifts (training on historical data, testing on recent years) and source-domain shifts (training on one SSP scenario, testing on others). This protocol assesses how well models generalize to changing climate dynamics and divergent emission pathways. It specifically measures performance degradation when distribution shifts occur in temporal or scenario domains. Use when the user wants to benchmark on ClimateSet, or asks about evaluating this task. Reports latitude-longitude weighted root mean squared error (RMSE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.12147
  bibtex_key: agana2025mlclimatedynamics
  confidence: high
---

# climate-set-ood-eval

> Do machine learning climate models work in changing climate dynamics? — Agana Navarro et al. (2025) (arXiv:2509.12147, 2025)

## What this evaluates

Evaluates the out-of-distribution robustness of machine learning climate emulation models under time-domain shifts (training on historical data, testing on recent years) and source-domain shifts (training on one SSP scenario, testing on others). This protocol assesses how well models generalize to changing climate dynamics and divergent emission pathways. It specifically measures performance degradation when distribution shifts occur in temporal or scenario domains.

## Datasets

- **ClimateSet** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `latitude-longitude weighted root mean squared error (RMSE)` **(primary)** — range: other
  - Root mean squared error between predicted and ground truth monthly surface air temperature and precipitation, weighted by latitude and longitude grid cells.

## Input / output format

**Input**: Climate forcing emission fields of CO2, CH4, BC, and SO2.

**Output**: Predictions of monthly surface air temperature and precipitation.

## Scoring recipe

```python
def compute_weighted_rmse(predictions, ground_truth, lat_weights, lon_weights):
    grid_weights = np.outer(lat_weights, lon_weights)
    grid_weights = grid_weights / grid_weights.sum()
    squared_errors = (predictions - ground_truth) ** 2
    weighted_mse = np.sum(grid_weights * squared_errors)
    return np.sqrt(weighted_mse)
```

## Common pitfalls

- Confusing the baseline evaluation protocol (test on SSP2-4.5) with the OOD protocols (time-domain or source-domain shifts), which use different train-test splits.
- Ignoring the latitude-longitude weighting in RMSE calculation, which is essential for climate grid data and differs from standard unweighted MSE.
- Assuming improved RMSE under time-domain shifts indicates general robustness, as source-domain shifts (different SSPs) often cause significant performance degradation.

## Evidence (verbatim from paper)

> We evaluate the model’s performance using the latitude-longitude weighted root mean squared error (RMSE), the metric reported from ClimateSet [[12]]. Predictions of monthly surface air temperature and precipitation are assessed against outputs from traditional climate models.

## Citation

```bibtex
@misc{agana2025mlclimatedynamics,
  title={Do machine learning climate models work in changing climate dynamics?},
  author={Agana Navarro et al. (2025)},
  year={2025},
  note={arXiv:2509.12147}
}
```

- arXiv: 2509.12147