Spatiotemporal Forecasting Eval

---
name: spatiotemporal-forecasting-eval
description: Evaluates spatiotemporal forecasting models on predicting future frames or climate indices from historical observations. It probes pixel-level reconstruction accuracy, structural similarity, and the model's ability to mitigate error propagation over extended lead times. Use when the user wants to benchmark on Moving MNIST, TrafficBJ, Human 3.6, SEVIR, ICAR-ENSO, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.20117
  bibtex_key: kim2024maskedautoregressive
  confidence: high
---

# spatiotemporal-forecasting-eval

> Masked Autoregressive Model for Weather Forecasting — Kim et al. (2024) (arXiv:2409.20117, 2024)

## What this evaluates

Evaluates spatiotemporal forecasting models on predicting future frames or climate indices from historical observations. It probes pixel-level reconstruction accuracy, structural similarity, and the model's ability to mitigate error propagation over extended lead times.

## Datasets

- **Moving MNIST** — total 20000; splits: train (10000), test (10000)
- **TrafficBJ** — total 20961; splits: train (19627), test (1334)
- **Human 3.6** — total 3759; splits: train (2624), test (1135)
- **SEVIR** — total 47877; splits: train (35718), test (12159)
- **ICAR-ENSO** — total 6872; splits: train (5205), test (1667)

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error between predicted and ground truth frames: (1/N) * Σ(pred - gold)^2.
- `MAE` — range: other
  - Mean Absolute Error between predicted and ground truth frames: (1/N) * Σ|pred - gold|.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index Measure evaluating luminance, contrast, and structure similarity between image windows.
- `CSI-M` — range: [0, 1]
  - Critical Success Index averaged over thresholds [16, 74, 133, 160, 181, 219]: CSI = Hits / (Hits + Misses + False Alarms) per threshold.
- `C-Nino3.4` — range: [-1, 1]
  - Correlation skill of the three-month-averaged Nino3.4 index (SST anomalies in 170°W-120°W, 5°S-5°N) over 12 forecasting steps.

## Input / output format

**Input**: Spatiotemporal tensor of shape (C, H, W, T) representing past observations (e.g., video frames, radar imagery, or SST anomalies).

**Output**: Spatiotemporal tensor of shape (C, H, W, T_hat) representing predicted future observations.

## Scoring recipe

```python
def compute_metrics(pred, gold):
    mse = np.mean((pred - gold) ** 2)
    mae = np.mean(np.abs(pred - gold))
    ssim = compute_ssim(pred, gold)
    thresholds = [16, 74, 133, 160, 181, 219]
    csi_scores = []
    for thr in thresholds:
        pred_bin = (pred > thr).astype(int)
        gold_bin = (gold > thr).astype(int)
        hits = np.sum((pred_bin == 1) & (gold_bin == 1))
        misses = np.sum((pred_bin == 0) & (gold_bin == 1))
        false_alarms = np.sum((pred_bin == 1) & (gold_bin == 0))
        csi = hits / (hits + misses + false_alarms) if (hits + misses + false_alarms) > 0 else 0
        csi_scores.append(csi)
    csi_m = np.mean(csi_scores)
    return {'MSE': mse, 'MAE': mae, 'SSIM': ssim, 'CSI-M': csi_m}
```

## Common pitfalls

- TrafficBJ metrics (MSE, MAE, SSIM) saturate due to linear past-future relationships, making them poor discriminators for this dataset.
- CSI-M requires averaging over six specific thresholds [16, 74, 133, 160, 181, 219]; using a single threshold or different values yields incorrect scores.
- ENSO evaluation (C-Nino3.4) requires three-month averaging of SST anomalies and correlation skill calculation over 12 forecasting steps, not raw pixel-wise MSE.

## Evidence (verbatim from paper)

> For the evaluation of common benchmark datasets, we adopt widely used evaluation metrics, including MSE, MAE, Peak Signal to Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM). For rain forecasting models, we use the Critical Success Index (CSI) as an evaluation metric [[38]]. In addition, we validate ENSO forecasting using the Nino SST indices [[16]]. Specifically, the Nino3.4 index represents the averaged SST anomalies across a specific Pacific region (170∘W-120∘W, 5∘S-5∘N), and defines El Niño/La Niña events based on the SST anomalies around the equator.

## Citation

```bibtex
@misc{kim2024maskedautoregressive,
  title={Masked Autoregressive Model for Weather Forecasting},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2409.20117}
}
```

- arXiv: 2409.20117