Time Series Forecasting Eval

---
name: time-series-forecasting-eval
description: Evaluates the capability of LLM-based models to forecast multivariate time series across multiple prediction horizons and real-world datasets. It probes pattern-aware temporal modeling and semantic alignment by measuring prediction accuracy under a channel-independent, rolling forecasting setup. Use when the user wants to benchmark on ETTh1, ETTh2, ETTm1, ETTm2, Weather, ECL, Traffic, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.07195
  bibtex_key: sun2025talon
  confidence: high
---

# time-series-forecasting-eval

> Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic Alignment — Yanru Sun et al. (arXiv:2508.07195, 2025)

## What this evaluates

Evaluates the capability of LLM-based models to forecast multivariate time series across multiple prediction horizons and real-world datasets. It probes pattern-aware temporal modeling and semantic alignment by measuring prediction accuracy under a channel-independent, rolling forecasting setup.

## Datasets

- **ETTh1** — total ?; splits: train/val/test (-1)
- **ETTh2** — total ?; splits: train/val/test (-1)
- **ETTm1** — total ?; splits: train/val/test (-1)
- **ETTm2** — total ?; splits: train/val/test (-1)
- **Weather** — total ?; splits: train/val/test (-1)
- **ECL** — total ?; splits: train/val/test (-1)
- **Traffic** — total ?; splits: train/val/test (-1)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error: the average of the squared differences between predicted and actual values. Formula: MSE = (1/n) Σ(y_true - y_pred)².
- `MAE` — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual values. Formula: MAE = (1/n) Σ|y_true - y_pred|.

## Input / output format

**Input**: Multivariate time series data, channel-independently decomposed into univariate sequences and segmented into non-overlapping patches of length S.

**Output**: Forecasted values for a specified prediction horizon (96, 192, 336, or 720 steps) generated autoregressively.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    mse = np.mean((y_true - y_pred) ** 2)
    mae = np.mean(np.abs(y_true - y_pred))
    return mse, mae
```

## Common pitfalls

- Rolling forecasting is used: a single model trained on a 96-step horizon is evaluated on longer horizons (192, 336, 720) without retraining.
- Channel-independent strategy is applied: each variable is modeled separately before aggregation, which may mask cross-variable dependencies.
- IMP score is reported as the average MSE and MAE reduction over baselines across seven datasets, not a direct model metric.

## Evidence (verbatim from paper)

> Multivariate forecasting (672-pred-{96, 192, 336, 720}) results under the one-for-all setting. Following (Liu et al. [2024c]), a single model is trained on a 96-step prediction horizon and evaluated on all horizons using rolling forecasting. The best results are in bold, and the second-best are underlined. Averaged results are reported here and full results are provided in Appendix. IMP denotes the average MSE and MAE reduction of TALON over each baseline across seven datasets.

## Citation

```bibtex
@misc{sun2025talon,
  title={Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic Alignment},
  author={Yanru Sun et al.},
  year={2025},
  note={arXiv:2508.07195}
}
```

- arXiv: 2508.07195