Dwrf Weather Forecast Eval

---
name: dwrf-weather-forecast-eval
description: Evaluates the accuracy, uncertainty quantification, and physical consistency of high-resolution ensemble weather forecasts for renewable energy applications. It probes a model's ability to downscale coarse atmospheric data to 1 km resolution while preserving multi-scale turbulence, thermodynamic constraints, and extreme event probabilities. Use when the user wants to benchmark on Northwestern Gobi Desert Wind Farm & ERA5 Reanalysis, or asks about evaluating this task. Reports RMSE, CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.04396
  bibtex_key: wang2025dwrf
  confidence: high
---

# dwrf-weather-forecast-eval

> Supporting renewable energy planning and operation with data-driven high-resolution ensemble weather forecast — Jingnan Wang et al. (2025) (arXiv:2505.04396, 2025)

## What this evaluates

Evaluates the accuracy, uncertainty quantification, and physical consistency of high-resolution ensemble weather forecasts for renewable energy applications. It probes a model's ability to downscale coarse atmospheric data to 1 km resolution while preserving multi-scale turbulence, thermodynamic constraints, and extreme event probabilities.

## Datasets

- **Northwestern Gobi Desert Wind Farm & ERA5 Reanalysis** — total ?; splits: train (-1), test (-1)

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Square Error: the square root of the mean of the squared differences between forecasted and observed values. Lower values indicate better point-forecast accuracy.
- `CRPS` **(primary)** — range: other
  - Continuous Ranked Probability Score: measures the accuracy of probabilistic forecasts by integrating the squared difference between the forecast cumulative distribution function and the observation's step function. Lower values indicate better probabilistic calibration.

## Input / output format

**Input**: Coarse-resolution (25 km) ERA5 atmospheric fields and global forecasts, conditioned on a learned climatological prior.

**Output**: High-resolution (1 km) ensemble forecasts (50 members) for multiple meteorological variables (wind components, temperature, humidity, sea level pressure) at 15-minute temporal resolution.

## Scoring recipe

```python
def compute_metrics(predictions, observations, ensemble_members=50):
    # predictions: shape (time, vars, h, w)
    # observations: shape (time, vars, h, w) or point stations
    rmse = np.sqrt(np.mean((predictions - observations) ** 2))
    
    crps_scores = []
    for t in range(time_steps):
        f_cdf = np.sort(ensemble_members[:, t]) / ensemble_members.shape[0]
        obs = observations[t]
        # Approximate CRPS for ensemble
        crps = np.mean(np.abs(f_cdf - (f_cdf >= obs).astype(float)))
        crps_scores.append(crps)
    return rmse, np.mean(crps_scores)
```

## Common pitfalls

- Dynamical models (WRF) suffer from spin-up issues causing initial forecast fluctuations that degrade early-time RMSE.
- Deterministic ML models (U-Net) and CGANs tend to underestimate extreme wind speeds and produce blurry spatial patterns due to MSE/loss objectives.
- Ensemble spread must be calibrated against observation coverage; wider spread does not always imply better skill if mean accuracy degrades.

## Evidence (verbatim from paper)

> We employed two key metrics for this evaluation: Root Mean Square Error (RMSE), which quantifies the average magnitude of forecast errors, and Continuous Ranked Probability Score (CRPS), which assesses the quality of probabilistic forecasts. The WRF simulation, driven by hourly ERA5 reanalysis data, served as our reference standard. Our analysis encompassed a wide range of meteorological variables, including both upper-level parameters (temperature and humidity at 500 and 700hPa) and surface variables (10-meter wind components U10m and V10m, sea level pressure SLP, and 2-meter temperature T2m).

## Citation

```bibtex
@misc{wang2025dwrf,
  title={Supporting renewable energy planning and operation with data-driven high-resolution ensemble weather forecast},
  author={Jingnan Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.04396}
}
```

- arXiv: 2505.04396