Urban Syn Uda Eval

---
name: urban-syn-uda-eval
description: Evaluates the utility of a semi-procedurally generated synthetic driving dataset (UrbanSyn) for unsupervised domain adaptation (UDA) in semantic segmentation. It probes whether combining multiple synthetic sources reduces the domain gap and improves pixel-level classification accuracy on real-world urban driving benchmarks. Use when the user wants to benchmark on UrbanSyn, GTAV, Synscapes, Cityscapes, BDD100K, Mapillary Vistas, or asks about evaluating this task. Reports self-labeling accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.12176
  bibtex_key: [REDACTED]
  confidence: high
---

# urban-syn-uda-eval

> All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes — Gómez et al. (2023) (arXiv:2312.12176, 2023)

## What this evaluates

Evaluates the utility of a semi-procedurally generated synthetic driving dataset (UrbanSyn) for unsupervised domain adaptation (UDA) in semantic segmentation. It probes whether combining multiple synthetic sources reduces the domain gap and improves pixel-level classification accuracy on real-world urban driving benchmarks.

## Datasets

- **UrbanSyn** — total ?; splits: train (-1)
- **GTAV** — total ?; splits: train (-1)
- **Synscapes** — total ?; splits: train (-1)
- **Cityscapes** — total ?; splits: test (-1)
- **BDD100K** — total ?; splits: test (-1)
- **Mapillary Vistas** — total ?; splits: test (-1)

## Metrics

- `self-labeling accuracy` **(primary)** — range: [0, 1]
  - Pixel-wise classification accuracy computed on real-world test images using pseudo-labels generated by the UDA model during co-training. Standard semantic segmentation UDA protocols typically report this as mean Intersection over Union (mIoU) across all classes.

## Input / output format

**Input**: RGB images of urban driving scenes. Training data consists of synthetic images with pixel-level ground truth (semantic masks, instance masks, depth, bounding boxes, occlusion). Test data consists of unlabeled real-world driving images.

**Output**: Per-pixel semantic class predictions for each input image.

## Scoring recipe

```python
def compute_accuracy(predictions, ground_truth):
    correct = np.sum(predictions == ground_truth)
    total = ground_truth.size
    return correct / total
# Note: In practice, this is typically aggregated per-class and averaged (mIoU) to handle class imbalance.
```

## Common pitfalls

- Assuming that simply concatenating multiple synthetic datasets automatically closes the domain gap without proper co-training or self-labeling strategies.
- Overlooking per-class performance disparities, as synthetic datasets may underrepresent rare classes (e.g., Bus, Rider) compared to real-world benchmarks.
- Confusing synthetic pixel-level ground truth with real-world annotations, which can lead to unrealistic evaluation expectations.

## Evidence (verbatim from paper)

> These labels are crucial for conducting synth-to-real UDA to develop semantic segmentation models without human labeling. We conduct comprehensive experiments in this section to demonstrate the importance of these labels. ... achieves state-of-the-art performance in unsupervised domain adaptation for semantic segmentation on real-world benchmarks (Cityscapes, BDD100K, Mapillary Vistas), demonstrating that diversity across synthetic sources outperforms single-dataset baselines by reducing domain gap and improving self-labeling accuracy via HRDA and co-training frameworks.

## Citation

```bibtex
@misc{gomez2023urbansyn,
  title={All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes},
  author={Gómez et al. (2023)},
  year={2023},
  note={arXiv:2312.12176}
}
```

- arXiv: 2312.12176