Chaotic Time Series Forecasting Eval

---
name: chaotic-time-series-forecasting-eval
description: Evaluates the ability of time series forecasting models to predict future values of noisy, chaotic dynamical systems. It probes how well models capture underlying nonlinear dynamics and handle varying levels of observation noise and system complexity. Use when the user wants to benchmark on Gilpin chaotic systems benchmark, or asks about evaluating this task. Reports SMAPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.13247
  bibtex_key: donner2023projected
  confidence: high
---

# chaotic-time-series-forecasting-eval

> A projected nonlinear state-space model for forecasting time series signals — Donner et al. (2023) (arXiv:2311.13247, 2023)

## What this evaluates

Evaluates the ability of time series forecasting models to predict future values of noisy, chaotic dynamical systems. It probes how well models capture underlying nonlinear dynamics and handle varying levels of observation noise and system complexity.

## Datasets

- **Gilpin chaotic systems benchmark** — total ?; splits: train (1000), test (200)

## Metrics

- `SMAPE` **(primary)** — range: percent
  - Symmetric Mean Absolute Percentage Error: 200 * mean(|z_t - z_hat_t| / (|z_t| + |z_hat_t|)), averaged over the forecasting period.

## Input / output format

**Input**: Noisy univariate time series observations, often constructed via delayed coordinates of dimension D_x.

**Output**: Forecasted time series values for the subsequent 200 time steps.

## Scoring recipe

```python
def compute_smape(z_true, z_pred):
    numerator = np.abs(z_true - z_pred)
    denominator = np.abs(z_true) + np.abs(z_pred)
    denominator = np.where(denominator == 0, 1, denominator)
    return 200 * np.mean(numerator / denominator)
```

## Common pitfalls

- Hyperparameters for baseline models (LSTM, Transformer, NBEATS) were optimized directly on the test dataset using SMAPE, which may inflate reported performance.
- The evaluation uses a fixed temporal split (first 1000 points for training, next 200 for testing) rather than a standard train/val/test partition, which can lead to high variance across systems with different lengths.

## Evidence (verbatim from paper)

> To noisy observations of the 127 chaotic dynamics we fitted fourteen different time series prediction methods (including the PNL-SS method) whose hyperparameters are optimized by test datasets using the symmetric mean absolute percent error (SMAPE) as an error measure. ... Here, the forecast error was measured by the symmetric mean absolute percentage error (SMAPE) defined as $200\langle{\left|z_{t}-\hat{z}_{t}\right|/(\left|z_{t}\right|+\left|\hat{z}_{t}\right|)}\rangle$, where $z_{t}$ is the test data and $\hat{z}_{t}$ is the forecasting values and $\langle\cdot\rangle$ denotes time average within a forecasting period (i.e., $t\=1001,\ldots,1200$).

## Citation

```bibtex
@misc{donner2023projected,
  title={A projected nonlinear state-space model for forecasting time series signals},
  author={Donner et al. (2023)},
  year={2023},
  note={arXiv:2311.13247}
}
```

- arXiv: 2311.13247