Time Moe Forecasting Eval

---
name: time-moe-forecasting-eval
description: Evaluates long-term time series forecasting capabilities of foundation models in both zero-shot (unseen datasets) and in-distribution (fine-tuned) settings across multiple prediction horizons. Use when the user wants to benchmark on ETTh1, ETTh2, ETTm1, ETTm2, Weather, Global Temp, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.16040
  bibtex_key: shi2024timemoe
  confidence: high
---

# time-moe-forecasting-eval

> Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts — Xiaoming Shi et al. (2024) (arXiv:2409.16040, 2024)

## What this evaluates

Evaluates long-term time series forecasting capabilities of foundation models in both zero-shot (unseen datasets) and in-distribution (fine-tuned) settings across multiple prediction horizons.

## Datasets

- **ETTh1** — total ?; splits: train (-1), test (-1)
- **ETTh2** — total ?; splits: train (-1), test (-1)
- **ETTm1** — total ?; splits: train (-1), test (-1)
- **ETTm2** — total ?; splits: train (-1), test (-1)
- **Weather** — total ?; splits: train (-1), test (-1)
- **Global Temp** — total ?; splits: train (-1), test (-1)

## Metrics

- `MSE` **(primary)** — range: [0, inf)
  - Mean Squared Error: average of squared differences between predicted and actual values. Lower is better.
- `MAE` — range: [0, inf)
  - Mean Absolute Error: average of absolute differences between predicted and actual values. Lower is better.

## Input / output format

**Input**: Time series sequences of fixed lengths (512, 1024, 2048, or 3072) corresponding to prediction horizons (96, 192, 336, or 720).

**Output**: Forecasted time series values for the specified prediction horizon.

## Scoring recipe

```python
def compute_metrics(pred, true):
    mse = np.mean((true - pred) ** 2)
    mae = np.mean(np.abs(true - pred))
    return {'MSE': mse, 'MAE': mae}
```

## Common pitfalls

- TimesFM is excluded from the Weather dataset evaluation due to pretraining data overlap.
- In-distribution (full-shot) evaluation strictly uses only one fine-tuning epoch.
- Input and output lengths are rigidly paired per horizon (e.g., 512 input for 96-step forecast).
- Reported averages in tables are computed across all six datasets and four horizons.

## Evidence (verbatim from paper)

> The evaluation metrics adopt mean square error (MSE) and mean absolute error (MAE).

## Citation

```bibtex
@misc{shi2024timemoe,
  title={Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts},
  author={Xiaoming Shi et al. (2024)},
  year={2024},
  note={arXiv:2409.16040}
}
```

- arXiv: 2409.16040