Chaos Forecasting Eval

---
name: chaos-forecasting-eval
description: Evaluates the ability of time series forecasting models to predict trajectories of low-dimensional chaotic dynamical systems. It probes how well models capture underlying deterministic chaos, smoothness, and multi-scale temporal dependencies without explicit trend or seasonality signals. Use when the user wants to benchmark on Chaotic Dynamical Systems Benchmark, or asks about evaluating this task. Reports sMAPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.05266
  bibtex_key: gilpin2021chaos
  confidence: high
---

# chaos-forecasting-eval

> Chaos as an interpretable benchmark for forecasting and data-driven modelling — Gilpin (2021) (arXiv:2110.05266, 2021)

## What this evaluates

Evaluates the ability of time series forecasting models to predict trajectories of low-dimensional chaotic dynamical systems. It probes how well models capture underlying deterministic chaos, smoothness, and multi-scale temporal dependencies without explicit trend or seasonality signals.

## Datasets

- **Chaotic Dynamical Systems Benchmark** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/williamgilpin/dysts

## Metrics

- `sMAPE` **(primary)** — range: percent
  - Symmetric Mean Absolute Percentage Error. Computed as the average of the absolute difference between forecast and actual values divided by their sum, scaled by 200 to express as a percentage. The paper notes it is reported as the headline metric for model comparison.
- `MSE` — range: other
  - Mean Squared Error. The average of the squared differences between predicted and actual values. Included as a secondary metric alongside MASE, MAE, MARRE, |CV|, 1-r², MAPE, and sMAPE.

## Input / output format

**Input**: Univariate or multivariate time series trajectories sampled from chaotic dynamical systems at specified granularities (coarse or fine). Models receive historical segments to generate forecasts for a specified prediction horizon.

**Output**: Predicted time series values for the target prediction horizon, aligned with the ground truth trajectory. Models are evaluated on unseen test trajectories emanating from different initial conditions than the training data.

## Scoring recipe

```python
def compute_smape(actual, predicted):
    import numpy as np
    actual = np.asarray(actual)
    predicted = np.asarray(predicted)
    numerator = np.abs(actual - predicted)
    denominator = np.abs(actual) + np.abs(predicted)
    denominator = np.where(denominator == 0, 1, denominator)
    error = 200 * np.mean(numerator / denominator)
    return error
```

## Common pitfalls

- Using chronological train/test splits instead of differential initial conditions; the benchmark explicitly requires training on one trajectory and testing on a separate trajectory starting from a different initial condition.
- Assuming traditional statistical models (e.g., Prophet, ARIMA) will dominate; the smooth, continuous nature of chaotic attractors favors deep learning architectures like Transformers and NBEATS.
- Ignoring granularity and noise variations; model rankings remain stable across two orders of magnitude in sampling granularity and varying noise, but absolute error scales significantly.

## Evidence (verbatim from paper)

> For all forecasting models and dynamical systems we compute eight error metrics: the mean squared error (MSE), mean absolute scaled error (MASE), mean absolute error (MAE), mean absolute ranged relative error (MARRE), the magnitude of the coefficient of variation  $(|CV|)$ , one minus the coefficient of determination  $(1 - r^2)$ , and the symmetric and regular mean absolute percent errors (MAPE and sMAPE). We find that all of these potential metrics are positively correlated across our dataset, and that they can be grouped into families of strongly-related metrics (Figure 2B). We also observe that the relative ranking of different forecasting models is independent of the choice of metric. Hereafter, we report sMAPE errors when comparing models, but we include all other metrics within the benchmark.

## Citation

```bibtex
@misc{gilpin2021chaos,
  title={Chaos as an interpretable benchmark for forecasting and data-driven modelling},
  author={Gilpin (2021)},
  year={2021},
  note={arXiv:2110.05266}
}
```

- arXiv: 2110.05266