Climax Letkf Eval

---
name: climax-letkf-eval
description: Evaluates the stability and error covariance representation of an AI-based weather prediction model (ClimaX) when integrated into an ensemble data assimilation system (LETKF). It probes the model's ability to generate physically consistent ensemble forecasts, capture flow-dependent error growth, and propagate observation information to unobserved variables without filter divergence. Use when the user wants to benchmark on WeatherBench, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.17781
  bibtex_key: kotsuki2024ensemble
  confidence: high
---

# climax-letkf-eval

> Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1 — Kotsuki et al. (2024) (arXiv:2407.17781, 2024)

## What this evaluates

Evaluates the stability and error covariance representation of an AI-based weather prediction model (ClimaX) when integrated into an ensemble data assimilation system (LETKF). It probes the model's ability to generate physically consistent ensemble forecasts, capture flow-dependent error growth, and propagate observation information to unobserved variables without filter divergence.

## Datasets

- **WeatherBench** — total ?; splits: test (-1)

## Metrics

- `RMSE` **(primary)** — range: other
  - Global-mean root mean square error computed over time for specific atmospheric variables (temperature, geopotential height, winds, humidity, surface pressure). Lower values indicate better forecast/analysis accuracy.
- `MAE_diff` — range: other
  - Mean absolute error difference between analysis and first-guess states relative to WeatherBench truth: MAE_diff = (1/N_t) Σ_t (|x_t^a - x_t^WB| - |x_t^b - x_t^WB|). Negative values indicate DA improvement, positive values indicate degradation.
- `Inflation factor (β)` — range: other
  - Covariance inflation factor estimated from observation-space statistics during the LETKF cycle. Used to diagnose error growth rates and chaotic behavior of the AI model.

## Input / output format

**Input**: Ensemble forecast states from ClimaX, observation data, and localization scale parameters (L_h) for the LETKF data assimilation cycle.

**Output**: Analysis states, first-guess (background) states, and computed error metrics (RMSE, MAE_diff) for each variable and localization scale configuration.

## Scoring recipe

```python
def evaluate(analysis_states, first_guess_states, truth_states):
    rmse = {}
    mae_diff = {}
    for var in variables:
        pred = analysis_states[var]
        gold = truth_states[var]
        bg = first_guess_states[var]
        rmse[var] = np.sqrt(np.mean((pred - gold)**2))
        mae_diff[var] = np.mean(np.abs(pred - gold)) - np.mean(np.abs(bg - gold))
    return rmse, mae_diff
```

## Common pitfalls

- Filter divergence at large localization scales (e.g., 800 km) stems from erroneous error covariance, not model failure.
- Improvements in observed variables (temperature) do not guarantee improvements in unobserved variables (geopotential height) or winds due to suboptimal spatial/inter-variable covariance.
- The inflation factor β diagnoses error growth dynamics rather than direct forecasting accuracy; lower β indicates slower error growth compared to dynamical models.

## Evidence (verbatim from paper)

> Figure 3 presents the time series of global-mean root mean square errors (RMSEs) for temperature and geopotential height at the fifth model level, with four different horizontal localization scales (L_h). After the initiation of data assimilation, all experiments showed reductions in analysis errors. ... Here, we investigate the spatial patterns of the difference between the analysis and first-guess mean absolute errors, which is given by: MAE_diff = (1/N_t) Σ_t |x_t^a - x_t^WB| - |x_t^b - x_t^WB|, where N_t is the sample size and superscript WB represents WeatherBench data.

## Citation

```bibtex
@misc{kotsuki2024ensemble,
  title={Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1},
  author={Kotsuki et al. (2024)},
  year={2024},
  note={arXiv:2407.17781}
}
```

- arXiv: 2407.17781