Toto Ts Forecasting Eval

---
name: toto-ts-forecasting-eval
description: Evaluates zero-shot and fine-tuned time series forecasting capabilities on real-world observability telemetry and general-purpose benchmarks. Probes model robustness to high-dimensional, nonstationary, multivariate series with skewed distributions and varying temporal intervals. Use when the user wants to benchmark on Boom, Boomlet, GIFT-Eval, LSF, or asks about evaluating this task. Reports CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14766
  bibtex_key: cohen2025observability
  confidence: high
---

# toto-ts-forecasting-eval

> This Time is Different: An Observability Perspective on Time Series Foundation Models — Ben Cohen et al. (2025) (arXiv:2505.14766, 2025)

## What this evaluates

Evaluates zero-shot and fine-tuned time series forecasting capabilities on real-world observability telemetry and general-purpose benchmarks. Probes model robustness to high-dimensional, nonstationary, multivariate series with skewed distributions and varying temporal intervals.

## Datasets

- **Boom** — total 350000000; splits: test (-1)
- **Boomlet** — total 23000000; splits: test (-1)
- **GIFT-Eval** — total 158000000; splits: test (-1)
- **LSF** — total 11000000; splits: train (-1), test (-1)

## Metrics

- `CRPS` **(primary)** — range: [0, ∞)
  - Continuous Ranked Probability Score. Normalized by the Seasonal Naive forecast baseline and aggregated across tasks using a geometric mean.
- `MASE` — range: [0, ∞)
  - Mean Absolute Scaled Error. Normalized by the Seasonal Naive forecast baseline and aggregated across tasks using a shifted geometric mean.
- `Rank` — range: [1, N]
  - Mean rank across all forecasting tasks with respect to CRPS. Lower is better.
- `MAE` — range: [0, ∞)
  - Mean Absolute Error. Reported per dataset and averaged across LSF tasks.
- `MSE` — range: [0, ∞)
  - Mean Squared Error. Reported per dataset and averaged across LSF tasks.

## Input / output format

**Input**: Historical multivariate time series data with varying intervals (10 sec to 1 day/yr) and series lengths (101 to 16,384 points). Input is processed via per-variate patch scaling.

**Output**: Forecasted values for a specified prediction length (48 to 900 points), output as point estimates or probabilistic distributions (Student-T mixture).

## Scoring recipe

```python
def aggregate_metrics(task_results):
    crps_vals = [r['crps'] for r in task_results.values()]
    mase_vals = [r['mase'] for r in task_results.values()]
    crps_agg = exp(mean(log(crps_vals)))
    mase_agg = exp(mean(log(mase_vals)))
    ranks = [rank_in_list(crps_vals, r['crps']) for r in task_results.values()]
    rank_agg = mean(ranks)
    return {'CRPS': crps_agg, 'MASE': mase_agg, 'Rank': rank_agg}
```

## Common pitfalls

- Data leakage in competing models on GIFT-Eval due to partial training data overlap with the benchmark.
- Inference speed and lack of batch support prevent some models (e.g., TabPFN) from running on large benchmarks like Boom.
- Full-shot deep learning models are often impractical at scale for observability data, making zero-shot comparison more relevant for this domain.

## Evidence (verbatim from paper)

> MASE and CRPS are normalized by the Seasonal Naive forecast and aggregated across tasks using shifted geometric mean. Rank is the mean rank across tasks with respect to CRPS.

## Citation

```bibtex
@misc{cohen2025observability,
  title={This Time is Different: An Observability Perspective on Time Series Foundation Models},
  author={Ben Cohen et al. (2025)},
  year={2025},
  note={arXiv:2505.14766}
}
```

- arXiv: 2505.14766