Cityscapes Eval

---
name: cityscapes-eval
description: Evaluates semantic scene understanding models on complex urban street scenes by measuring pixel-level classification accuracy and instance-level segmentation quality. It probes the model's ability to handle high-resolution imagery, diverse weather/lighting conditions, and fine-grained class distinctions in autonomous driving contexts. Use when the user wants to benchmark on Cityscapes, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1604.01685
  bibtex_key: cordts2016cityscapes
  confidence: high
---

# cityscapes-eval

> The Cityscapes Dataset for Semantic Urban Scene Understanding — Cordts et al. (2016) (arXiv:1604.01685, 2016)

## What this evaluates

Evaluates semantic scene understanding models on complex urban street scenes by measuring pixel-level classification accuracy and instance-level segmentation quality. It probes the model's ability to handle high-resolution imagery, diverse weather/lighting conditions, and fine-grained class distinctions in autonomous driving contexts.

## Datasets

- **Cityscapes** — total ?; splits: train (-1), val (-1)

## Metrics

- `IoU` **(primary)** — range: [0, 1]
  - Intersection over Union: the ratio of the area of overlap between the predicted mask and the ground truth mask to the area of their union. Computed per class and typically averaged to mIoU.
- `iIoU` — range: [0, 1]
  - Instance-normalized IoU, which adjusts the standard IoU calculation to account for instance-level variations and class imbalance in dense urban scenes.

## Input / output format

**Input**: Single-frame monocular LDR images, often resized, cropped, or split into halves depending on the baseline's memory constraints.

**Output**: Per-pixel semantic class labels (dense masks) or instance-level segmentation masks/bounding boxes.

## Scoring recipe

```python
def compute_iou(pred_mask, gt_mask, class_id):
    pred = (pred_mask == class_id)
    gt = (gt_mask == class_id)
    intersection = np.logical_and(pred, gt).sum()
    union = np.logical_or(pred, gt).sum()
    if union == 0:
        return 1.0
    return intersection / union

def evaluate(dataset, model):
    ious = []
    for img, gt in dataset:
        pred = model(img)
        for cls in range(num_classes):
            ious.append(compute_iou(pred, gt, cls))
    return np.mean(ious)  # mIoU
```

## Common pitfalls

- Ignoring void pixels during training causes gradients to be induced incorrectly, degrading performance.
- Input resolution and cropping strategies vary significantly across baselines, making direct comparison of inference time and accuracy difficult without normalization.
- Coarse annotations are used for pretraining or weak supervision, which can artificially inflate performance if not properly accounted for in the evaluation protocol.

## Evidence (verbatim from paper)

> Tables 9 and 11 list all individual class-level IoU scores for all control experiments and baselines. Tables 10 and 12 give the corresponding instance-normalized iIoU scores.

## Citation

```bibtex
@misc{cordts2016cityscapes,
  title={The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author={Cordts et al. (2016)},
  year={2016},
  note={arXiv:1604.01685}
}
```

- arXiv: 1604.01685