Climate Ood Robustness Eval

---
name: climate-ood-robustness-eval
description: Evaluates the out-of-distribution robustness of climate emulators under temporal extrapolation and cross-scenario forcing shifts. It probes whether models trained on historical climate data can accurately generalize to novel future regimes and extreme emission pathways without seeing them during training. Use when the user wants to benchmark on ClimateSet / CMIP6 GCM outputs, or asks about evaluating this task. Reports LL-RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.23043
  bibtex_key: [REDACTED]
  confidence: high
---

# climate-ood-robustness-eval

> Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts — Agana Navarro et al. (2026) (arXiv:2603.23043, 2026)

## What this evaluates

Evaluates the out-of-distribution robustness of climate emulators under temporal extrapolation and cross-scenario forcing shifts. It probes whether models trained on historical climate data can accurately generalize to novel future regimes and extreme emission pathways without seeing them during training.

## Datasets

- **ClimateSet / CMIP6 GCM outputs** — total ?; splits: train (-1), test_temporal (-1), test_SSP1-2.6 (-1), test_SSP5-8.5 (-1)

## Metrics

- `LL-RMSE` **(primary)** — range: [0, ∞)
  - Root Mean Squared Error computed on log-transformed or link-function-transformed target variables (surface air temperature and precipitation). Lower values indicate better accuracy.

## Input / output format

**Input**: Spatial-temporal climate data including forcing agents and atmospheric responses for surface air temperature (TAS) and precipitation (PR) from GCMs (EC-Earth3, MPI-ESM1-2-HR).

**Output**: Predicted values for TAS and PR.

## Scoring recipe

```python
def compute_ll_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_pred - y_true) ** 2))

def compute_percent_change(ll_rmse_id, ll_rmse_oob):
    return ((ll_rmse_oob - ll_rmse_id) / ll_rmse_id) * 100
```

## Common pitfalls

- High absolute LL-RMSE values (0.8–1.1) are noted as a limitation due to standardized training without per-scenario hyperparameter tuning, which may mask intrinsic architectural fragility but ensures fair comparison.
- Potential data contamination from ClimaX's pre-training on CMIP6 historical data blurs the boundary between in-distribution and out-of-distribution evaluation.
- Precipitation shows consistently higher degradation than temperature due to stochastic hydrological physics, meaning temperature-only evaluations may overestimate model robustness.

## Evidence (verbatim from paper)

> The absolute LL-RMSE values achieved (ranging from 0.8 to 1.1) represent significantly lower accuracy than state-of-the-art emulators, which typically achieve errors in the 0.2–0.3 range. In contrast, Precipitation (PR) saw a consistent performance degradation across all architectures, with LL-RMSE increasing by up to 5.18%.

## Citation

```bibtex
@misc{agana_navarro2026climateood,
  title={Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts},
  author={Agana Navarro et al. (2026)},
  year={2026},
  note={arXiv:2603.23043}
}
```

- arXiv: 2603.23043