Power System Forecasting Eval

---
name: power-system-forecasting-eval
description: Evaluates the zero-shot and fine-tuning performance of time-series foundation models and deep learning baselines on deterministic and probabilistic power system forecasting tasks. It probes capabilities including horizon sensitivity, multivariate covariate handling, and generalization to unseen geographic sites. Use when the user wants to benchmark on ARPA-E PERFORM, or asks about evaluating this task. Reports nMAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22077
  bibtex_key: zater2026empirical
  confidence: high
---

# power-system-forecasting-eval

> Empirical Assessment of Time-Series Foundation Models For Power System Forecasting Applications — Za'ter et al. (2026) (arXiv:2604.22077, 2026)

## What this evaluates

Evaluates the zero-shot and fine-tuning performance of time-series foundation models and deep learning baselines on deterministic and probabilistic power system forecasting tasks. It probes capabilities including horizon sensitivity, multivariate covariate handling, and generalization to unseen geographic sites.

## Datasets

- **ARPA-E PERFORM** — total ?; splits: train (-1), test (-1)

## Metrics

- `nMAE` **(primary)** — range: percent
  - Normalized Mean Absolute Error, calculated as the mean absolute error divided by the mean of the test set target values, expressed as a percentage.
- `nRMSE` — range: percent
  - Normalized Root Mean Squared Error, calculated as the root mean squared error divided by the mean of the test set target values, expressed as a percentage.
- `CRPS` — range: percent
  - Continuous Ranked Probability Score, measuring the accuracy of probabilistic forecasts by integrating the squared difference between the predicted cumulative distribution function and the actual outcome.

## Input / output format

**Input**: Univariate or multivariate time series sequences (historical target values, optionally with weather covariates) for specific solar/wind sites or load zones. Evaluated at 60-minute, 6-hour, and 24-hour look-ahead horizons.

**Output**: Point forecasts (single scalar per timestep) for deterministic tasks, or full predictive distributions (e.g., quantiles or mixture parameters) for probabilistic tasks.

## Scoring recipe

```python
def compute_nmae(preds, targets, target_mean):
    return np.mean(np.abs(preds - targets)) / target_mean * 100

def compute_nrmse(preds, targets, target_mean):
    return np.sqrt(np.mean((preds - targets)**2)) / target_mean * 100

def compute_crps(pred_cdf, actual):
    # pred_cdf: array of predicted CDF values at actual, actual: scalar
    return np.mean((pred_cdf - actual)**2)
```

## Common pitfalls

- Assuming zero-shot foundation model performance is operationally viable; the paper shows nMAE >7% is too high for grid operations without fine-tuning.
- Ignoring the normalization denominator for nMAE/nRMSE; values are percentages relative to the test-set mean, not absolute errors.
- Failing to separate seen vs. unseen sites in evaluation; generalization gaps are location-specific and require strict spatial train/test splits.

## Evidence (verbatim from paper)

> Throughout these experiments, solar and wind forecasting are evaluated on a per-site basis, while load forecasting is evaluated on a per-zone basis. Furthermore, all reported performance metrics are averaged across their respective sites or zones. The magnitude of the nMAE and nRMSE values indicates that current foundation models, when applied directly without adaptation, struggle to capture the domain-specific dynamics of renewable generation and demand despite some of the model pre-training data including energy applications.

## Citation

```bibtex
@misc{zater2026empirical,
  title={Empirical Assessment of Time-Series Foundation Models For Power System Forecasting Applications},
  author={Za'ter et al. (2026)},
  year={2026},
  note={arXiv:2604.22077}
}
```

- arXiv: 2604.22077