Caf 7m Eval

---
name: caf-7m-eval
description: Evaluates whether context-aided forecasting models can effectively leverage supplementary descriptive context to improve probabilistic time series forecasts, and tests generalization to out-of-domain real-world datasets. Use when the user wants to benchmark on CAF-7M, CGTSF, GIFT-Eval, or asks about evaluating this task. Reports CRPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.12451
  bibtex_key: zheng2026overcoming
  confidence: high
---

# caf-7m-eval

> Overcoming the Modality Gap in Context-Aided Forecasting — Zheng et al. (2026) (arXiv:2603.12451, 2026)

## What this evaluates

Evaluates whether context-aided forecasting models can effectively leverage supplementary descriptive context to improve probabilistic time series forecasts, and tests generalization to out-of-domain real-world datasets.

## Datasets

- **CAF-7M** — total ?; splits: HARD (-1), EASY (-1), ALL (-1); repo https://github.com/ServiceNow/DoubleCast
- **CGTSF** — total ?; splits: test (-1)
- **GIFT-Eval** — total ?; splits: test (-1)

## Metrics

- `CRPS` **(primary)** — range: other
  - Continuous Ranked Probability Score, a proper scoring rule for probabilistic forecasts. Normalized by dividing the mean CRPS by the mean absolute value of the ground truth to prevent large values from dominating aggregation.
- `Win Rate` — range: [0, 1]
  - The ratio of evaluation windows where the model's CRPS is lower than Chronos' CRPS. Designed to be robust to score outliers.

## Input / output format

**Input**: Historical time series values paired with descriptive textual context. Evaluated via zero-shot prompting or fine-tuned model inference.

**Output**: Probabilistic forecast distribution over the prediction horizon.

## Scoring recipe

```python
def compute_crps(pred_dist, actual):
    return proper_scoring_rule(pred_dist, actual)

def normalize_crps(crps_values, actuals):
    return np.mean(crps_values) / np.mean(np.abs(actuals))

def compute_win_rate(model_crps, baseline_crps):
    return np.mean(model_crps < baseline_crps)
```

## Common pitfalls

- Using unverified synthetic contexts can falsely attribute poor performance to model inability rather than low context quality.
- Auto-regressive chunking on prediction lengths exceeding pretraining limits (e.g., >64 steps) causes distribution mismatch, skewing zero-shot results.
- Context swapping/omission ablations are required to verify actual context utilization, as models may ignore or misalign with provided text.

## Evidence (verbatim from paper)

> Since Direct Prompt produces a probabilistic forecast, we use the CRPS (gneiting2007strictly) as our primary metric, which is a proper scoring rule for probabilistic forecasts. To avoid large values dominating the aggregated results, we normalize the CRPS using the mean absolute values of the forecast ground truths. As a complementary metric that is robust to score outliers, we also compute the Win Rate: the ratio of windows where Direct Prompt has a lower CRPS than Chronos.

## Citation

```bibtex
@misc{zheng2026overcoming,
  title={Overcoming the Modality Gap in Context-Aided Forecasting},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.12451}
}
```

- arXiv: 2603.12451