Time Series Benchmark Eval

---
name: time-series-benchmark-eval
description: Evaluates zero-shot forecasting performance of time series foundation models across diverse datasets and horizons. Probes model capability to capture structural temporal patterns (trend, seasonality, stationarity, complexity) and generalizes to unseen data without leakage. Use when the user wants to benchmark on TIME Benchmark, or asks about evaluating this task. Reports MASE, CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12147
  bibtex_key: qiao2026time
  confidence: high
---

# time-series-benchmark-eval

> It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks — Qiao et al. (2026) (arXiv:2602.12147, 2026)

## What this evaluates

Evaluates zero-shot forecasting performance of time series foundation models across diverse datasets and horizons. Probes model capability to capture structural temporal patterns (trend, seasonality, stationarity, complexity) and generalizes to unseen data without leakage.

## Datasets

- **TIME Benchmark** — total 50; splits: test (-1)

## Metrics

- `MASE` **(primary)** — range: [0, inf)
  - Mean Absolute Scaled Error; measures point forecast accuracy by scaling the mean absolute error against the mean absolute error of a seasonal naive baseline.
- `CRPS` **(primary)** — range: [0, inf)
  - Continuous Ranked Probability Score; measures probabilistic forecast accuracy by integrating the squared difference between the predicted cumulative distribution function and the actual outcome.

## Input / output format

**Input**: Historical time series window (univariate or multivariate) with a specified forecasting horizon.

**Output**: Point forecasts (single value per horizon step) or probabilistic forecasts (distribution samples or quantiles).

## Scoring recipe

```python
# 1. Compute raw metric per evaluation unit (task/variate)
raw_metric = compute_metric(predictions, ground_truth)
baseline_metric = compute_metric(seasonal_naive_predictions, ground_truth)

# 2. Normalize against baseline
norm_metric = raw_metric / baseline_metric

# 3. Aggregate across all units using geometric mean
final_score = geometric_mean(norm_metric_across_units)

# Note: For distribution-based models, sample 100 quantiles prior to metric computation.
```

## Common pitfalls

- Metrics are normalized against a Seasonal Naive baseline; raw scores are not directly comparable across datasets.
- Geometric mean is used for aggregation, not arithmetic mean, to handle multiplicative relationships and outliers.
- Distribution-based models require sampling 100 quantiles prior to metric computation to match quantile-based models.

## Evidence (verbatim from paper)

> For metrics, we employ MASE and CRPS for point and probabilistic evaluation, respectively. ... we adopt the protocol in (Aksu et al., [2024]) and utilize a consistent relative evaluation framework where model metrics are normalized against a Seasonal Naive (S-Naive) baseline.

## Citation

```bibtex
@misc{qiao2026time,
  title={It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks},
  author={Qiao et al. (2026)},
  year={2026},
  note={arXiv:2602.12147}
}
```

- arXiv: 2602.12147