Us Grid Forecasting Eval

---
name: us-grid-forecasting-eval
description: Evaluates the ability of deep learning architectures (SSMs, Transformers, RNNs) to forecast hourly electricity load across major US power grids. It probes how well models capture temporal patterns, handle varying prediction horizons, and integrate exogenous weather covariates for accurate grid-scale forecasting. Use when the user wants to benchmark on US ISO Hourly Load Data (EIA-930), or asks about evaluating this task. Reports MSE (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.21415
  bibtex_key: hong2026benchmarking
  confidence: high
---

# us-grid-forecasting-eval

> Benchmarking State Space Models, Transformers, and Recurrent Networks for US Grid Forecasting — Hong et al. (2026) (arXiv:2602.21415, 2026)

## What this evaluates

Evaluates the ability of deep learning architectures (SSMs, Transformers, RNNs) to forecast hourly electricity load across major US power grids. It probes how well models capture temporal patterns, handle varying prediction horizons, and integrate exogenous weather covariates for accurate grid-scale forecasting.

## Datasets

- **US ISO Hourly Load Data (EIA-930)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/gramm-ai/grid-forecast-benchmark

## Metrics

- `MSE (%)` **(primary)** — range: percent
  - Normalized root mean squared error expressed as a percentage of the test set mean load: 100 / y_bar * sqrt(1/n * sum((y_i - y_hat_i)^2)).
- `MAPE (%)` — range: percent
  - Mean absolute percentage error: 100 / n * sum(|(y_i - y_hat_i) / y_i|).
- `nMAE` — range: percent
  - Normalized mean absolute error used when actuals approach zero: 100 / (n * y_bar) * sum(|y_i - y_hat_i|).

## Input / output format

**Input**: Sequence of hourly load values (and optionally weather covariates: temperature, humidity, wind speed, GHI, cloud cover) over a 240-hour context window, with temporal features (hour-of-day, day-of-week) encoded as learnable embeddings.

**Output**: Predicted load values for a specified forecast horizon W (24, 48, 72, 96, or 168 hours).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    y_bar = np.mean(y_true)
    mse_pct = 100.0 / y_bar * np.sqrt(np.mean((y_true - y_pred)**2))
    mape_pct = 100.0 / len(y_true) * np.mean(np.abs((y_true - y_pred) / y_true))
    return {'MSE (%)': mse_pct, 'MAPE (%)': mape_pct}
```

## Common pitfalls

- MAPE becomes numerically unstable when actual load values approach zero; the authors switch to normalized MAE (nMAE) for generation/price signals to avoid division by near-zero values.
- Weather-integration experiments use a rolling-origin walk-forward backtest and observed reanalysis data rather than NWP forecasts, making their absolute MAPE values incomparable to the fixed-split load-only benchmark.
- Z-score normalization must strictly use training-set statistics only to prevent data leakage into validation/test sets.

## Evidence (verbatim from paper)

> Our primary metric is mean squared error, reported as a normalized percentage MSE (%) for cross-grid comparability, alongside mean absolute percentage error MAPE (%): where $ar{y}$ is the mean load of the test set. MSE (%) normalizes the root mean squared error by mean load, enabling direct comparison across grids of different scale.

## Citation

```bibtex
@misc{hong2026benchmarking,
  title={Benchmarking State Space Models, Transformers, and Recurrent Networks for US Grid Forecasting},
  author={Hong et al. (2026)},
  year={2026},
  note={arXiv:2602.21415}
}
```

- arXiv: 2602.21415