Piano Pde Weather Eval

---
name: piano-pde-weather-eval
description: Evaluates the capability of physics-informed autoregressive models to accurately forecast time-dependent partial differential equations and global atmospheric variables over multi-step horizons. Use when the user wants to benchmark on PDE Benchmarks (Wave, Reaction, Convection, Heat), ERA5, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.16235
  bibtex_key: nagda2025piano
  confidence: high
---

# piano-pde-weather-eval

> PIANO: Physics Informed Autoregressive Network — Nagda et al. (2025) (arXiv:2508.16235, 2025)

## What this evaluates

Evaluates the capability of physics-informed autoregressive models to accurately forecast time-dependent partial differential equations and global atmospheric variables over multi-step horizons.

## Datasets

- **PDE Benchmarks (Wave, Reaction, Convection, Heat)** — total ?; splits: test (-1)
- **ERA5** — total ?; splits: test (-1)

## Metrics

- `rMAE` — range: percent
  - Relative Mean Absolute Error: the mean absolute difference between predicted and ground truth states, normalized by the magnitude of the ground truth.
- `rRMSE` — range: percent
  - Relative Root Mean Squared Error: the square root of the mean squared difference between predictions and ground truth, normalized by the ground truth magnitude.
- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error: the square root of the average squared differences between predicted and observed values. For weather forecasting, it is latitude-weighted to account for spherical geometry.
- `ACC` — range: [-1, 1]
  - Anomaly Correlation Coefficient: measures the correlation between predicted and observed anomalies (deviations from the mean), capturing directional accuracy.

## Input / output format

**Input**: Discretized spatio-temporal state grids at initial or previous time steps, spatial gradients, velocity fields, and spatiotemporal embeddings.

**Output**: Predicted spatio-temporal state u(x,t) and velocity field v(t) at the target time step.

## Scoring recipe

```python
def compute_metrics(pred, true, lat):
    # PDE relative errors
    rMAE = mean(abs(pred - true)) / mean(abs(true))
    rRMSE = sqrt(mean((pred - true)**2)) / sqrt(mean(true**2))
    
    # Weather metrics (latitude-weighted)
    weights = cos(lat) / mean(cos(lat))
    RMSE = sqrt(mean(weights * (pred - true)**2))
    ACC = cov(pred - mean(pred), true - mean(true)) / (std(pred) * std(true))
    return rMAE, rRMSE, RMSE, ACC
```

## Common pitfalls

- Baseline papers do not report variance, so statistical significance testing was not performed.
- Weather metrics require latitude-weighting to correctly account for the spherical geometry of the Earth.
- PDE benchmarks use relative errors (normalized by ground truth), not absolute errors, which changes how performance scales with solution magnitude.

## Evidence (verbatim from paper)

> Performance is measured using relative Mean Absolute Error (rMAE) and relative Root Mean Squared Error (rRMSE), which are standard metrics in the PINN literature (Xu et al. [2025]). ... Performance is evaluated using two standard metrics: root mean square error (RMSE) and anomaly correlation coefficient (ACC). RMSE quantifies the absolute prediction error, while ACC measures the correlation between predicted and observed anomalies, capturing the directional accuracy. Both metrics are latitude-weighted to reflect the spherical geometry of the Earth.

## Citation

```bibtex
@misc{nagda2025piano,
  title={PIANO: Physics Informed Autoregressive Network},
  author={Nagda et al. (2025)},
  year={2025},
  note={arXiv:2508.16235}
}
```

- arXiv: 2508.16235