Tsfm Scaling Eval

---
name: tsfm-scaling-eval
description: Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks. Use when the user wants to benchmark on Monash subset, LSF subset, or asks about evaluating this task. Reports NLL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12360
  bibtex_key: yao2024towards
  confidence: high
---

# tsfm-scaling-eval

> Towards Neural Scaling Laws for Time Series Foundation Models — Yao et al. (2024) (arXiv:2410.12360, 2024)

## What this evaluates

Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks.

## Datasets

- **Monash subset** — total ?; splits: test (-1)
- **LSF subset** — total ?; splits: test (-1)

## Metrics

- `NLL` **(primary)** — range: other
  - Negative log-likelihood of the true values under the model's predictive distribution. Lower is better.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: average of |(actual - predicted) / actual| over the forecasting horizon.
- `SMAPE` — range: percent
  - Symmetric Mean Absolute Percentage Error: 2*|actual - predicted| / (|actual| + |predicted|), averaged over the horizon.
- `MASE` — range: other
  - Mean Absolute Scaled Error: ratio of MAE to the in-sample naive forecast MAE.
- `CRPS` — range: other
  - Continuous Ranked Probability Score: integrated squared difference between the predictive CDF and the step function at the observed value.

## Input / output format

**Input**: Historical time series sequences (context window) provided to the foundation model for zero-shot or fine-tuned forecasting.

**Output**: Point forecasts or full predictive distributions (probability density functions) for the target forecasting horizon.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, y_pred_dist=None):
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    smape = 2 * np.mean(np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred))) * 100
    mase = np.mean(np.abs(y_true - y_pred)) / np.mean(np.abs(y_true[1:] - y_true[:-1]))
    nll = -np.mean(y_pred_dist.log_prob(y_true)) if y_pred_dist else None
    crps = np.mean((y_pred_dist.cdf(y_true) - 0.5)**2) if y_pred_dist else None
    return {'MAPE': mape, 'SMAPE': smape, 'MASE': mase, 'NLL': nll, 'CRPS': crps}
```

## Common pitfalls

- OOD performance degrades predictably but scaling gains remain proportional across distributions, so absolute OOD scores should not be compared directly to ID scores.
- Different metrics exhibit distinct power-law exponents; scaling laws are metric-dependent and cannot be assumed to transfer across MAPE, NLL, etc.
- Performance improvements are not always smooth; 'emergent behaviors' cause abrupt, non-continuous jumps at specific model sizes (e.g., 10M parameters).

## Evidence (verbatim from paper)

> We investigate the scaling behaviors of five common performance metrics: NLL, MAPE, SMAPE, MASE, and CRPS, (as shown in Figures 15 - 19). All metrics exhibit a decreasing trend following an approximate power-law; however, each metric demonstrates distinct scaling characteristics, reflected in their varying power-law exponents.

## Citation

```bibtex
@misc{yao2024towards,
  title={Towards Neural Scaling Laws for Time Series Foundation Models},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2410.12360}
}
```

- arXiv: 2410.12360