Climatebench M Eval

---
name: climatebench-m-eval
description: Evaluates AI models on multi-modal climate data across three tasks: tensor time-series forecasting, extreme weather anomaly detection, and crop classification from satellite imagery. It probes spatial-temporal alignment, generative data synthesis, and robustness to highly imbalanced rare events. Use when the user wants to benchmark on ClimateBench-M, or asks about evaluating this task. Reports Mean Absolute Error (MAE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.07394
  bibtex_key: fu2025climatebenchm
  confidence: high
---

# climatebench-m-eval

> ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method — Fu et al. (2025) (arXiv:2504.07394, 2025)

## What this evaluates

Evaluates AI models on multi-modal climate data across three tasks: tensor time-series forecasting, extreme weather anomaly detection, and crop classification from satellite imagery. It probes spatial-temporal alignment, generative data synthesis, and robustness to highly imbalanced rare events.

## Datasets

- **ClimateBench-M** — total ?; splits: train (-1), test (-1); repo https://github.com/iDEA-iSAIL-Lab-UIUC/ClimateBench-M

## Metrics

- `Accuracy (Acc)` — range: [0, 1]
  - Evaluates the overlap between prediction and ground-truth: Acc = a/b, where a is the number of correct predictions and b is the total number of samples.
- `Mean Absolute Error (MAE)` **(primary)** — range: [0, inf)
  - Assesses the difference between prediction and ground truth. Standard MAE calculation over forecasted tensor values.
- `Intersection of Union (IoU)` — range: [0, 1]
  - Measures the ratio of the intersection of two sets over their union: IoU = |A∩B| / |A∪B|, where A is the prediction set and B is the ground-truth set.
- `Area Under the Receiver Operating Characteristic Curve (AUC-ROC)` — range: [0, 1]
  - Quantifies the ability to distinguish between classes by measuring the area under the ROC curve: AUC-ROC = ∫ b(a) da, where a and b are TPR and FPR respectively.

## Input / output format

**Input**: Forecasting: past 24-hour tensor time series. Anomaly Detection: hourly forecasted weather features at specific locations. Crop Classification: concatenated satellite images across time channels for a given location.

**Output**: Forecasting: future 24-hour tensor time series. Anomaly Detection: anomaly score/condition based on feature-wise generation probability. Crop Classification: predicted crop class or segmentation mask.

## Scoring recipe

```python
def compute_metrics(pred, gold):
    acc = np.sum(pred == gold) / len(gold)
    mae = np.mean(np.abs(pred - gold))
    intersection = len(set(pred) & set(gold))
    union = len(set(pred) | set(gold))
    iou = intersection / union if union > 0 else 0.0
    fpr, tpr, _ = roc_curve(gold, pred)
    auc = np.trapz(tpr, fpr)
    return {'Acc': acc, 'MAE': mae, 'IoU': iou, 'AUC-ROC': auc}
```

## Common pitfalls

- Baselines for anomaly detection use ground-truth observations while SGM uses forecast features, creating an asymmetric evaluation setup.
- Tensor time-series baselines are flattened to multi-variate series, with results averaged across all 238 locations.
- Crop segmentation baselines lack native temporal modeling, requiring manual channel concatenation of multi-temporal images.
- Extreme weather anomalies are highly rare (~0.45% frequency), making standard supervised baselines prone to poor performance.

## Evidence (verbatim from paper)

> We measure the performance of the baseline methods as well as the proposed method on the ClimateBench-M with respect to the following metrics: (1) Accuracy (Acc): It evaluates the overlap between the prediction and the ground-truth, i.e., Acc = a/b, where a is the number of correct prediction and b is the total number of samples. (2) Mean Absolute Error (MAE): It assess the difference between the prediction and the ground truth... (3) Intersection of Union (IoU): It measures the ratio of the intersection of two sets over the union of two sets... (4) Area Under the Receiver Operating Characteristic Curve (AUC-ROC): It quantifies the ability of a model to distinguish between classes by measuring the area under the ROC curve.

## Citation

```bibtex
@misc{fu2025climatebenchm,
  title={ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method},
  author={Fu et al. (2025)},
  year={2025},
  note={arXiv:2504.07394}
}
```

- arXiv: 2504.07394