Wildscenes Eval

---
name: wildscenes-eval
description: Probes the capability of models to perform semantic segmentation on unstructured, large-scale natural environments using both 2D images and 3D LiDAR point clouds. It evaluates robustness to semantic ambiguity, clutter, and temporal environmental shifts in outdoor traversals. Use when the user wants to benchmark on WildScenes, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.15364
  bibtex_key: [REDACTED]
  confidence: high
---

# wildscenes-eval

> WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments — Vidanapathirana et al. (2023) (arXiv:2312.15364, 2023)

## What this evaluates

Probes the capability of models to perform semantic segmentation on unstructured, large-scale natural environments using both 2D images and 3D LiDAR point clouds. It evaluates robustness to semantic ambiguity, clutter, and temporal environmental shifts in outdoor traversals.

## Datasets

- **WildScenes** — total ?; splits: train (6051), val (283), test (2133)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Standard Mean Intersection over Union computed over 15 classes for 2D segmentation and 12 classes for 3D segmentation. IoU for a class is the intersection of predicted and ground truth pixels/points divided by their union.

## Input / output format

**Input**: 2D: RGB images (cropped to 512x512). 3D: LiDAR point clouds.

**Output**: Per-pixel (2D) or per-point (3D) semantic class labels from a predefined set of 15 (2D) or 12 (3D) classes.

## Scoring recipe

```python
def compute_miou(preds, targets, num_classes):
    ious = []
    for c in range(num_classes):
        pred_mask = (preds == c)
        gt_mask = (targets == c)
        intersection = np.sum(pred_mask & gt_mask)
        union = np.sum(pred_mask | gt_mask)
        ious.append(intersection / union if union > 0 else 0.0)
    return np.mean(ious)
```

## Common pitfalls

- 3D evaluation excludes 'sky', 'water', and 'other-terrain' classes due to LiDAR limitations or insufficient training points, resulting in 12 classes instead of the full 15.
- Some traversals are allocated to buffer regions and excluded from the train/val/test splits, meaning the reported split sizes do not cover the entire dataset.
- Backbones are initialized with ImageNet-1k/22k weights, which may not reflect zero-shot performance on out-of-domain natural scenes.

## Evidence (verbatim from paper)

> For evaluating the performance of a semantic segmentation method with respect to the ground truth label annotations, we use the standard Mean Intersection over Union (mIoU) metric for a set of 15 classes (2D) and 12 classes (3D).

## Citation

```bibtex
@misc{vidanapathirana2023wildscenes,
  title={WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments},
  author={Vidanapathirana et al. (2023)},
  year={2023},
  note={arXiv:2312.15364}
}
```

- arXiv: 2312.15364