Probabilistic Ts Forecasting Eval

---
name: probabilistic-ts-forecasting-eval
description: Evaluates real-time probabilistic forecasting of financial and weather time series, probing a model's ability to quantify uncertainty via quantile modeling and maintain calibration over sequential submission rounds. Use when the user wants to benchmark on DAX, Wind, Temperature, or asks about evaluating this task. Reports skill score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.16171
  bibtex_key: bracher2022learning
  confidence: high
---

# probabilistic-ts-forecasting-eval

> Learning to forecast: The probabilistic time series forecasting challenge — Bracher et al. (2022) (arXiv:2211.16171, 2022)

## What this evaluates

Evaluates real-time probabilistic forecasting of financial and weather time series, probing a model's ability to quantify uncertainty via quantile modeling and maintain calibration over sequential submission rounds.

## Datasets

- **DAX, Wind, Temperature** — total 199; splits: test (199); repo https://github.com/FK83/ptsfc_replication

## Metrics

- `skill score` **(primary)** — range: other
  - Computed relative to a benchmark forecast using a quantile scoring function. A positive skill score indicates that the forecaster outperforms the benchmark.
- `coverage rate` — range: percent
  - The share of test-sample observations that fall within a specified prediction interval (e.g., 50% or 95% level).

## Input / output format

**Input**: Historical time series data and optional external feature variables for targets (DAX, wind speed, temperature).

**Output**: Probabilistic forecasts in the form of prediction intervals or quantiles at multiple forecast horizons (e.g., 1-7 days for DAX, 36-84 hours for weather).

## Scoring recipe

```python
def compute_skill_score(predictions, gold, benchmark):
    forecaster_score = quantile_scoring_function(predictions, gold)
    benchmark_score = quantile_scoring_function(benchmark, gold)
    skill = benchmark_score - forecaster_score
    return skill
```

## Common pitfalls

- Evaluating on a static train/test split instead of the required real-time sequential submission process.
- Confusing calibration (coverage rates) with sharpness (interval width); both are needed for good probabilistic forecasts.
- Small sample sizes per horizon (13-14 weeks) can lead to noisy coverage estimates.

## Evidence (verbatim from paper)

> Figure 3 summarizes performance as measured by the quantile scoring function, based on which we compute skill scores relative to the benchmark forecasts (see Equation 1). A positive skill score indicates that a forecast outperforms the benchmark.

## Citation

```bibtex
@misc{bracher2022learning,
  title={Learning to forecast: The probabilistic time series forecasting challenge},
  author={Bracher et al. (2022)},
  year={2022},
  note={arXiv:2211.16171}
}
```

- arXiv: 2211.16171