Timer Time Series Eval

---
name: timer-time-series-eval
description: Evaluates a large decoder-only Transformer model on standard time series benchmarks for forecasting, imputation, and anomaly detection. It probes the model's few-shot generalization, scalability, and robustness in data-scarce scenarios compared to encoder-only baselines. Use when the user wants to benchmark on ETT, ECL, Traffic, Weather, PEMS, UCR Anomaly Archive, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.02368
  bibtex_key: liu2024timer
  confidence: high
---

# timer-time-series-eval

> Timer: Generative Pre-trained Transformers Are Large Time Series Models — Liu et al. (2024) (arXiv:2402.02368, 2024)

## What this evaluates

Evaluates a large decoder-only Transformer model on standard time series benchmarks for forecasting, imputation, and anomaly detection. It probes the model's few-shot generalization, scalability, and robustness in data-scarce scenarios compared to encoder-only baselines.

## Datasets

- **ETT** — total ?; splits: test (-1)
- **ECL** — total ?; splits: test (-1)
- **Traffic** — total ?; splits: test (-1)
- **Weather** — total ?; splits: test (-1)
- **PEMS** — total ?; splits: test (-1)
- **UCR Anomaly Archive** — total ?; splits: test (-1)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error between predicted and ground truth values, calculated as the average of squared differences.
- `MAE` — range: other
  - Mean Absolute Error between predicted and ground truth values, calculated as the average of absolute differences.

## Input / output format

**Input**: Time series data tokenized into a unified single-series sequence (S3) format. Forecasting uses a fixed lookback window of 672 time points; imputation uses segments of length 24 with some fully masked; anomaly detection uses observed segments to predict the next segment.

**Output**: Autoregressive next-token predictions corresponding to the forecasted, imputed, or anomalous segment values.

## Scoring recipe

```python
def compute_metrics(pred, true):
    mse = np.mean((pred - true) ** 2)
    mae = np.mean(np.abs(pred - true))
    return mse, mae
```

## Common pitfalls

- Strictly exclude downstream datasets from the pre-training corpus (UTSD) to prevent data leakage.
- Maintain the fixed lookback length of 672 and forecast length of 96 for forecasting tasks.
- Replicate few-shot scenarios using exact data scarcity ratios (1%, 5%, 20%, 100%) rather than full training sets.

## Evidence (verbatim from paper)

> We adopt the unified lookback length as $672$ and the forecast length as $96$. We pre-training Timer on UTSD-12G with the segment length $S\=96$ and the number of tokens $N\=15$, such that Timer can deal with time series with the context length up to $1440$. And the downstream forecasting task is naturally converted into the next token prediction as detailed in Appendix[B.2]. Table 2: In-domain and out-of-domain forecasting results by pre-training on the source domain and fine-tuning on the target dataset under different data scarcity. ... Metric | MSE | MAE

## Citation

```bibtex
@misc{liu2024timer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2402.02368}
}
```

- arXiv: 2402.02368