Climate Segmentation Eval

---
name: climate-segmentation-eval
description: This evaluation probes pixel-level weather pattern segmentation (atmospheric rivers and tropical cyclones) from multi-channel climate data. It measures both segmentation accuracy and exascale training throughput/scaling efficiency across different network architectures and hardware configurations. Use when the user wants to benchmark on Climate weather pattern dataset, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.01993
  bibtex_key: kurth2018exascale
  confidence: high
---

# climate-segmentation-eval

> Exascale Deep Learning for Climate Analytics — Kurth et al. (2018) (arXiv:1810.01993, 2018)

## What this evaluates

This evaluation probes pixel-level weather pattern segmentation (atmospheric rivers and tropical cyclones) from multi-channel climate data. It measures both segmentation accuracy and exascale training throughput/scaling efficiency across different network architectures and hardware configurations.

## Datasets

- **Climate weather pattern dataset** — total ?; splits: validation (-1)

## Metrics

- `IoU` **(primary)** — range: [0, 1]
  - Intersection over Union: the area of overlap between the predicted segmentation mask and the ground truth mask divided by the area of union between them.
- `Sustained Throughput` — range: PF/s
  - Training performance measured in petaflops per second (PF/s) or samples per second, calculated from floating-point operations per image and sustained training rate per GPU.
- `Parallel Efficiency` — range: percent
  - Ratio of actual scaled throughput to ideal linear scaling throughput, expressed as a percentage.

## Input / output format

**Input**: 16-channel climate data images (e.g., integrated water vapor) representing weather patterns.

**Output**: Pixel-level segmentation masks classifying atmospheric features (e.g., atmospheric rivers in blue, tropical cyclones in red).

## Scoring recipe

```python
def compute_iou(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union == 0:
        return 1.0
    return intersection / union
```

## Common pitfalls

- The weighted loss function penalizes false negatives on tropical cyclones ~37x more than false positives, which can inflate IoU by overpredicting features.
- FP16 training shows lower compute efficiency than FP32 due to memory-bound kernels, despite higher theoretical throughput.
- Scaling experiments without local data staging suffer significant efficiency drops (>9.5%) at large node counts due to filesystem bandwidth limits.

## Evidence (verbatim from paper)

> Segmentation accuracy is often measured using the intersection over union (IoU) metric. The Tiramisu network obtained an IoU of 59% on our validation data set, while our modified DeepLabv3+ network was able to achieve 73% IoU.

## Citation

```bibtex
@misc{kurth2018exascale,
  title={Exascale Deep Learning for Climate Analytics},
  author={Kurth et al. (2018)},
  year={2018},
  note={arXiv:1810.01993}
}
```

- arXiv: 1810.01993