Motime Eval

---
name: motime-eval
description: Evaluates multimodal time series forecasting models across two scenarios: varying-history forecasting (using long and short temporal sequences) and cold-start forecasting (predicting from minimal initial observations). It probes how well models leverage static external modalities like text and metadata to improve prediction accuracy, especially for sparse or short series. Use when the user wants to benchmark on PixelRec, AmazonReview, WikiPeople, Movielens, TaobaoFashion, Tianchi, News, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15072
  bibtex_key: zhou2025motime
  confidence: high
---

# motime-eval

> MoTime: A Dataset Suite for Multimodal Time Series Forecasting — Xin Zhou et al. (2025) (arXiv:2505.15072, 2025)

## What this evaluates

Evaluates multimodal time series forecasting models across two scenarios: varying-history forecasting (using long and short temporal sequences) and cold-start forecasting (predicting from minimal initial observations). It probes how well models leverage static external modalities like text and metadata to improve prediction accuracy, especially for sparse or short series.

## Datasets

- **PixelRec** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **AmazonReview** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **WikiPeople** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **Movielens** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **TaobaoFashion** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **Tianchi** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **News** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2). Captures error in the original scale and is sensitive to large deviations.
- `WRMSPE` — range: other
  - Weighted Root Mean Squared Percentage Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2) / (1/T * sum_{t=1}^T |y_t|). Normalizes RMSE by the mean absolute value of the ground truth for a scale-invariant view.

## Input / output format

**Input**: Historical time series values (7 steps for daily datasets, 6 steps for high-frequency News), optionally augmented with static external modalities (text, metadata, images) depending on the model variant.

**Output**: Forecasted time series values for a specified horizon (7 to 28 steps ahead for daily datasets, up to 12 steps ahead for News).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    T = len(y_true)
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    wrmspe = np.sqrt(np.mean((y_true - y_pred) ** 2)) / np.mean(np.abs(y_true))
    return rmse, wrmspe
```

## Common pitfalls

- Do not normalize or scale the data before evaluation; scores must be computed on the original data scale to preserve error meaningfulness.
- Avoid using scaled metrics like RMSSE, as the test spans are long enough for absolute-scale metrics and RMSSE is difficult to interpret with varying forecast horizons.
- In cold-start forecasting, use only 7 previous daily steps or 6 previous 20-minute steps from relevant non-target entities, not the target entity itself.

## Evidence (verbatim from paper)

> We report two widely used metrics [18], $$ \mathrm {R M S E} = \sqrt {\frac {1}{T} \sum_ {t = 1} ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}, \quad \mathrm {W R M S P E} = \frac{\sqrt {\frac {1}{T} \sum t = 1 ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}}{\frac {1}{T} \sum t = 1 ^ {T} | y _ {t} |} $$ RMSE captures error in the original scale and is particularly sensitive to large deviations. WRMSPE normalizes RMSE by the mean absolute value of the ground truth, offering a scale-invariant view of forecasting quality.

## Citation

```bibtex
@misc{zhou2025motime,
  title={MoTime: A Dataset Suite for Multimodal Time Series Forecasting},
  author={Xin Zhou et al. (2025)},
  year={2025},
  note={arXiv:2505.15072}
}
```

- arXiv: 2505.15072