Coralscapes Eval

---
name: coralscapes-eval
description: Probes semantic segmentation models on complex underwater scenes characterized by high morphological variability, degradation states, and visual distortions. It evaluates the model's ability to generalize across geographically distinct reef sites and handle severe class imbalance and fine-grained benthic classification. Use when the user wants to benchmark on Coralscapes, or asks about evaluating this task. Reports mean Intersection over Union (mIoU).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.20000
  bibtex_key: sauder2025coralscapes
  confidence: high
---

# coralscapes-eval

> The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs — Sauder et al. (2025) (arXiv:2503.20000, 2025)

## What this evaluates

Probes semantic segmentation models on complex underwater scenes characterized by high morphological variability, degradation states, and visual distortions. It evaluates the model's ability to generalize across geographically distinct reef sites and handle severe class imbalance and fine-grained benthic classification.

## Datasets

- **Coralscapes** — total 2075; splits: train (1517), val (166), test (392); HF `EPFL-ECEO/coralscapes`; repo https://github.com/eceo-epfl/coralscapesScripts

## Metrics

- `mean Intersection over Union (mIoU)` **(primary)** — range: [0, 1]
  - For each of the 39 benthic classes, compute IoU = TP / (TP + FP + FN). Average the IoU across all classes to obtain the final score. This is the standard metric for Cityscapes-mimic segmentation benchmarks.

## Input / output format

**Input**: 8-bit PNG images resized to 1024×2048px resolution, representing underwater reef scenes.

**Output**: Pixel-wise segmentation masks assigning each pixel to one of 39 benthic classes (e.g., live coral, dead coral, substrate, background).

## Scoring recipe

```python
def compute_miou(pred_masks, gt_masks, num_classes=39):
    miou_scores = []
    for cls in range(num_classes):
        pred_cls = (pred_masks == cls)
        gt_cls = (gt_masks == cls)
        intersection = np.logical_and(pred_cls, gt_cls).sum()
        union = np.logical_or(pred_cls, gt_cls).sum()
        if union == 0:
            miou_scores.append(1.0)
        else:
            miou_scores.append(intersection / union)
    return np.mean(miou_scores)
```

## Common pitfalls

- Spatial train-test split prevents models from memorizing site-specific features; they must generalize across geographically distinct reefs.
- Conservative annotation protocol means ambiguous or degraded corals are labeled as 'background' or 'dark', requiring models to handle high uncertainty and avoid speculative predictions.
- Severe class imbalance (rarest class has >400x fewer polygons than the most common) can bias models toward majority classes if not properly weighted.

## Evidence (verbatim from paper)

> We split the dataset spatially by reef site to allow a fair evaluation, resulting in a training set of 1517 images (27 sites), a validation set of 166 images (3 sites), and a test set of 392 images (5 sites).

## Citation

```bibtex
@misc{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder et al. (2025)},
  year={2025},
  note={arXiv:2503.20000}
}
```

- arXiv: 2503.20000