Cloudops Forecasting Eval

---
name: cloudops-forecasting-eval
description: Evaluates time series forecasting models, particularly pre-trained Transformers, on cloud operations data. It probes zero-shot generalization, architectural efficiency, and scaling behavior against classical and deep learning baselines. Use when the user wants to benchmark on azure2017, borg2011, ali2018, or asks about evaluating this task. Reports sMAPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.05063
  bibtex_key: woo2023pushing
  confidence: high
---

# cloudops-forecasting-eval

> Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain — Woo et al. (2023) (arXiv:2310.05063, 2023)

## What this evaluates

Evaluates time series forecasting models, particularly pre-trained Transformers, on cloud operations data. It probes zero-shot generalization, architectural efficiency, and scaling behavior against classical and deep learning baselines.

## Datasets

- **azure2017** — total ?; splits: test (-1)
- **borg2011** — total ?; splits: test (-1)
- **ali2018** — total ?; splits: test (-1)

## Metrics

- `sMAPE` **(primary)** — range: percent
  - Symmetric Mean Absolute Percentage Error. Computed as the average of |y - ŷ| / ((|y| + |ŷ|)/2) over all time steps. Lower is better.
- `CRPS` — range: other
  - Continuous Ranked Probability Score, measuring the accuracy of probabilistic forecasts against observed values. Lower is better.

## Input / output format

**Input**: Time series context window of length L=480 containing target values and covariates.

**Output**: Probabilistic forecast distribution parameters (e.g., Student-T parameters) for the prediction horizon.

## Scoring recipe

```python
def compute_smape(y_true, y_pred):
    diff = np.abs(y_true - y_pred)
    scale = (np.abs(y_true) + np.abs(y_pred)) / 2
    return np.mean(diff / scale) * 100

def compute_crps(y_true, dist_params):
    # CRPS for Student-T distribution
    return crps_student_t(y_true, dist_params)
```

## Common pitfalls

- Fine-tuning pre-trained models often yields no benefit over zero-shot inference due to sufficient pre-training diversity.
- Date/time features are not critical for forecasting in this domain; RoPE positional encodings perform better.
- The naive forecast serves as a surprisingly strong baseline for high-frequency CloudOps data.

## Evidence (verbatim from paper)

> We observe that our zero-shot approach constitutes a very strong baseline, obtaining a 27/24% reduction in sMAPE/CRPS from the next best performing method on the largest dataset, azure2017, generally outperforming all other methods.

## Citation

```bibtex
@misc{woo2023pushing,
  title={Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain},
  author={Woo et al. (2023)},
  year={2023},
  note={arXiv:2310.05063}
}
```

- arXiv: 2310.05063