India Weather Bench Eval

---
name: india-weather-bench-eval
description: Evaluates data-driven regional weather forecasting models over India under varying boundary conditioning strategies. It probes the ability of architectures to accurately predict multi-variable meteorological fields at high resolution and assesses their robustness during extreme weather events like heatwaves. Use when the user wants to benchmark on IndiaWeatherBench, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.00653
  bibtex_key: nguyen2025indiaweatherbench
  confidence: high
---

# india-weather-bench-eval

> IndiaWeatherBench: A Dataset and Benchmark for Data-Driven Regional Weather Forecasting over India — Nguyen et al. (2025) (arXiv:2509.00653, 2025)

## What this evaluates

Evaluates data-driven regional weather forecasting models over India under varying boundary conditioning strategies. It probes the ability of architectures to accurately predict multi-variable meteorological fields at high resolution and assesses their robustness during extreme weather events like heatwaves.

## Datasets

- **IndiaWeatherBench** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/tung-nd/IndiaWeatherBench

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error computed over the spatial grid (256x256), meteorological variables, and forecast lead times. Lower values indicate better accuracy.

## Input / output format

**Input**: A 39-channel tensor of shape (256, 256, 39) containing regional meteorological variables (e.g., 2m temperature, 10m wind, MSLP, 7 pressure levels). Optionally concatenated with boundary forcing (10-pixel ring of ground truth) or coarse-resolution ERA5 conditioning (124x124 interpolated to 256x256).

**Output**: A predicted 39-channel tensor of shape (256, 256, 39) representing the regional weather state at the target forecast time step.

## Scoring recipe

```python
def compute_rmse(pred, gold):
    # pred, gold: (batch, 256, 256, 39)
    diff = pred - gold
    mse = np.mean(diff ** 2, axis=(0, 1, 2, 3))
    return np.sqrt(mse).mean()
```

## Common pitfalls

- Architectures may fail when boundary conditioning strategy mismatches their input tokenization (e.g., Stormer degrades with coarse-resolution conditioning due to scale mixing in patches).
- Deterministic RMSE alone may mask systematic biases during extreme events; models can show low average error but large warm/cold biases in heatwaves.

## Evidence (verbatim from paper)

> We use RMSE as the evaluation metric, and refer readers to Appendix[10.2] for additional metrics.

## Citation

```bibtex
@misc{nguyen2025indiaweatherbench,
  title={IndiaWeatherBench: A Dataset and Benchmark for Data-Driven Regional Weather Forecasting over India},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.00653}
}
```

- arXiv: 2509.00653