Gift Eval Eval

---
name: gift-eval-eval
description: Evaluates zero-shot time series forecasting models across diverse domains, frequencies, and prediction horizons. It probes a model's ability to generalize to unseen multivariate and univariate series, identifying strengths and weaknesses in short-term versus long-term forecasting. Use when the user wants to benchmark on GIFT-Eval, or asks about evaluating this task. Reports MAPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.10393
  bibtex_key: aksu2024gift_eval
  confidence: high
---

# gift-eval-eval

> GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation — Aksu et al. (2024) (arXiv:2410.10393, 2024)

## What this evaluates

Evaluates zero-shot time series forecasting models across diverse domains, frequencies, and prediction horizons. It probes a model's ability to generalize to unseen multivariate and univariate series, identifying strengths and weaknesses in short-term versus long-term forecasting.

## Datasets

- **GIFT-Eval** — total 144000; splits: test (-1); repo https://github.com/SalesforceAIResearch/gift-eval

## Metrics

- `MAPE` **(primary)** — range: percent
  - Median of the Mean Absolute Percentage Error across all series. Computed as median(mean(|(y_true - y_pred) / y_true|) * 100). Normalized against the Seasonal Naive baseline.
- `CRPS` — range: other
  - Continuous Ranked Probability Score evaluating the accuracy of probabilistic forecasts against observed values. Normalized against the Seasonal Naive baseline.

## Input / output format

**Input**: Historical univariate or multivariate time series data across 7 domains, 10 frequencies, and varying prediction lengths.

**Output**: Point forecasts (single values) and/or probabilistic forecasts (full predictive distributions or samples).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, y_pred_dist, baseline_pred, baseline_dist):
    mape = median(mean(abs((y_true - y_pred) / y_true)) * 100)
    crps = continuous_ranked_probability_score(y_true, y_pred_dist)
    baseline_mape = median(mean(abs((y_true - baseline_pred) / y_true)) * 100)
    baseline_crps = continuous_ranked_probability_score(y_true, baseline_dist)
    norm_mape = mape / baseline_mape
    norm_crps = crps / baseline_crps
    return norm_mape, norm_crps
```

## Common pitfalls

- Pre-training datasets of foundation models (TimesFM, Chronos, Moirai) exhibit partial data leakage into GIFT-Eval, requiring careful isolation or retraining.
- Decoder-only foundation models use recursive multi-step forecasting, causing severe error accumulation for medium/long-term predictions.
- Metrics must be normalized against the Seasonal Naive baseline to standardize comparisons across different benchmarks.

## Evidence (verbatim from paper)

> Performance is assessed using two metrics: the median Mean Absolute Percentage Error (MAPE) for point forecasts and the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, 2007) for probabilistic forecasts (definition of both metrics are in Appendix C). To standardize comparison across benchmarks, both metrics are normalized against the Seasonal Naive baseline.

## Citation

```bibtex
@misc{aksu2024gift_eval,
  title={GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation},
  author={Aksu et al. (2024)},
  year={2024},
  note={arXiv:2410.10393}
}
```

- arXiv: 2410.10393