Quito Bench Eval

---
name: quito-bench-eval
description: Evaluates time series forecasting models on a regime-balanced benchmark stratified by trend, seasonality, and forecastability. It probes a model's ability to handle varying context lengths, forecast horizons, and multivariate dependencies while mitigating domain bias and information leakage. Use when the user wants to benchmark on QuitoBench, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.26017
  bibtex_key: xue2026quito
  confidence: high
---

# quito-bench-eval

> QuitoBench: A High-Quality Open Time Series Forecasting Benchmark — Xue et al. (2026) (arXiv:2603.26017, 2026)

## What this evaluates

Evaluates time series forecasting models on a regime-balanced benchmark stratified by trend, seasonality, and forecastability. It probes a model's ability to handle varying context lengths, forecast horizons, and multivariate dependencies while mitigating domain bias and information leakage.

## Datasets

- **QuitoBench** — total ?; splits: train (-1), val (-1), test (1290); repo https://github.com/alipay/quito

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and actual values over all time steps and series.

## Input / output format

**Input**: A contiguous window of a time series of length L (96, 576, or 1024) used to predict the next H steps (48, 288, or 512). Input is either multivariate (all 5 channels jointly) or univariate (each channel independently).

**Output**: A sequence of H predicted values corresponding to the forecast horizon.

## Scoring recipe

```python
def compute_mae(predictions, targets):
    return np.mean(np.abs(predictions - targets))

def compute_mean_rank(all_series_maes):
    # Rank models 1-10 per series/config, then average ranks
    ranks = rankdata(all_series_maes, method='min')
    return np.mean(ranks)
```

## Common pitfalls

- Using non-overlapping windows (stride H) instead of dense rolling windows (unit stride), which reduces evaluation instances by orders of magnitude and inflates variance.
- Averaging raw MAE across series with different scales without converting to ranks, which biases results toward high-magnitude series.
- Treating multivariate and univariate modes identically; MV predicts all channels jointly while UV processes them independently, leading to different performance rankings.

## Evidence (verbatim from paper)

> Metrics. We report MAE as the primary metric and convert per-series MAE to rank scores (1–10) to aggregate fairly across heterogeneous scales; the mean rank is reported over all series and configurations (See [section E.3] for details).

## Citation

```bibtex
@misc{xue2026quito,
  title={QuitoBench: A High-Quality Open Time Series Forecasting Benchmark},
  author={Xue et al. (2026)},
  year={2026},
  note={arXiv:2603.26017}
}
```

- arXiv: 2603.26017