Synn Air Pollution Forecasting Eval

---
name: synn-air-pollution-forecasting-eval
description: Evaluates a hybrid neural forecasting framework's ability to predict regional particulate matter (PM1, PM2.5, PM10) concentrations. It probes both average forecasting accuracy across spatial grids and the model's capacity to capture rare, high-impact pollution spikes and extreme events. Use when the user wants to benchmark on ERA5 & CAMS, or asks about evaluating this task. Reports Latitude-Weighted RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.23977
  bibtex_key: abeysinghe2025synn
  confidence: high
---

# synn-air-pollution-forecasting-eval

> Synergistic Neural Forecasting of Air Pollution with Stochastic Sampling — Abeysinghe et al. (2025) (arXiv:2510.23977, 2025)

## What this evaluates

Evaluates a hybrid neural forecasting framework's ability to predict regional particulate matter (PM1, PM2.5, PM10) concentrations. It probes both average forecasting accuracy across spatial grids and the model's capacity to capture rare, high-impact pollution spikes and extreme events.

## Datasets

- **ERA5 & CAMS** — total ?; splits: train (-1), test (-1)

## Metrics

- `Latitude-Weighted RMSE` **(primary)** — range: other
  - Root Mean Square Error adjusted by latitude weights to correct for regional area disparities and avoid unfair advantages for local models during spatial averaging.
- `RQE` — range: other
  - Relative Quantile Error measuring deviation at high quantiles (90th–99.99th percentiles) to emphasize accuracy in capturing extremes. Negative values indicate underestimation.
- `SEDI` — range: [0, 1]
  - Symmetric Extremal Dependency Index capturing the skill in classifying extreme vs. non-extreme events, providing robustness across different percentile thresholds.

## Input / output format

**Input**: Multi-variable meteorological (mslp, u10, v10, t2m) and PM (PM1, PM2.5, PM10) time-series data from ERA5/CAMS, spatially regridded to 0.4° or 5.26° resolution, with temporal context from previous timesteps.

**Output**: Predicted PM concentrations (PM1, PM2.5, PM10) in μg/m³ at specified spatial resolutions and lead times (1–6 days).

## Scoring recipe

```python
import numpy as np
# Latitude-Weighted RMSE
lat_weights = np.cos(np.deg2rad(lats))
lat_weights /= lat_weights.sum()
rmse = np.sqrt(np.mean(lat_weights * (pred - gold)**2))
# RQE
q = [0.90, 0.99, 0.9999]
rqe = np.mean((np.quantile(pred, q) - np.quantile(gold, q)) / np.quantile(gold, q))
# SEDI
thresh = np.quantile(gold, 0.90)
tp = np.sum((pred > thresh) & (gold > thresh))
fp = np.sum((pred > thresh) & (gold <= thresh))
fn = np.sum((pred <= thresh) & (gold > thresh))
tn = np.sum((pred <= thresh) & (gold <= thresh))
sed = 2 * (tp*tn - fp*fn) / ((tp+fp)*(fp+tn) + (tp+fn)*(fn+tn))
```

## Common pitfalls

- Spatial averaging without latitude weighting unfairly advantages local models due to longitudinal span disparities.
- Comparing models at their native resolutions without harmonizing via regridding (e.g., ClimaX protocol) leads to invalid performance comparisons.
- Focusing solely on RMSE masks the diffusion module's true value, which is primarily reflected in tail-sensitive metrics like RQE and SEDI.

## Evidence (verbatim from paper)

> Metrics: We use three metrics to quantify performance, with a focus on both general accuracy and extreme event prediction (see Appendix C for formulations). Latitude-Weighted RMSE: Assesses forecast accuracy across spatial locations, correcting for regional area disparities. We adjust the longitudinal averaging to reflect reduced latitude span, avoiding unfair advantage for local models. RQE: Measures deviation at high quantiles (90th–99.99th percentiles), emphasizing the accuracy in capturing extremes. Negative values indicate underestimation. SEDI: Captures the skill in classifying extreme vs. non-extreme events, providing robustness across different percentile thresholds.

## Citation

```bibtex
@misc{abeysinghe2025synn,
  title={Synergistic Neural Forecasting of Air Pollution with Stochastic Sampling},
  author={Abeysinghe et al. (2025)},
  year={2025},
  note={arXiv:2510.23977}
}
```

- arXiv: 2510.23977