Ts Forecasting Eval

---
name: ts-forecasting-eval
description: Evaluates the forecasting accuracy and computational efficiency of deep learning models on multivariate time series data. It probes how architectural choices, preprocessing steps, and spatial-temporal processing configurations impact performance across varying forecasting horizons. Use when the user wants to benchmark on Weather, Solar-Energy, ECL, Traffic, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.22702
  bibtex_key: moretti2025whatmatters
  confidence: high
---

# ts-forecasting-eval

> What Matters in Deep Learning for Time Series Forecasting? — Moretti et al. (2025) (arXiv:2512.22702, 2025)

## What this evaluates

Evaluates the forecasting accuracy and computational efficiency of deep learning models on multivariate time series data. It probes how architectural choices, preprocessing steps, and spatial-temporal processing configurations impact performance across varying forecasting horizons.

## Datasets

- **Weather** — total ?; splits: (unstated)
- **Solar-Energy** — total ?; splits: (unstated)
- **ECL** — total ?; splits: (unstated)
- **Traffic** — total ?; splits: (unstated)

## Metrics

- `MSE` — range: other
  - Mean Squared Error: the average of the squared differences between predicted and actual values. Formula: (1/N) * Σ(y_true - y_pred)².
- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual values. Formula: (1/N) * Σ|y_true - y_pred|.

## Input / output format

**Input**: Multivariate time series windows of configurable length, optionally augmented with exogenous covariates.

**Output**: Predicted values for the next `horizon` time steps for each variable in the series.

## Scoring recipe

```python
def compute_mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

def compute_mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)
```

## Common pitfalls

- Performance gains are often attributed to novel architectural components rather than implementation choices like channel-independence or attention across channels.
- Computational efficiency metrics (batch time, GPU memory) are reported but frequently ignored when comparing model rankings.
- Forecasting horizon length drastically changes model rankings; results at H=96 do not generalize to H=720.

## Evidence (verbatim from paper)

> The results in [Tab. 8] extend[Tab. 1] for both MSE and MAE. We added in[Tab. 9], a comparison of all the possible configurations—local, global, and hybrid—for linear models. The results are consistent with the findings discussed in[Sec. 4.1].

## Citation

```bibtex
@misc{moretti2025whatmatters,
  title={What Matters in Deep Learning for Time Series Forecasting?},
  author={Moretti et al. (2025)},
  year={2025},
  note={arXiv:2512.22702}
}
```

- arXiv: 2512.22702