Valerie22 Eval

---
name: valerie22-eval
description: This protocol evaluates the perceptual fidelity and cross-domain generalization capability of the VALERIE22 synthetic urban dataset by training a semantic segmentation model on it and testing on real-world automotive datasets. It specifically probes how dataset diversity (unique 3D assets) and training scale affect downstream perception performance. Use when the user wants to benchmark on VALERIE22, Cityscapes, A2D2, BDD100K, India Driving Dataset, Mapillary Vistas, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.09632
  bibtex_key: grau2023valerie22
  confidence: high
---

# valerie22-eval

> VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments — Grau et al. (2023) (arXiv:2308.09632, 2023)

## What this evaluates

This protocol evaluates the perceptual fidelity and cross-domain generalization capability of the VALERIE22 synthetic urban dataset by training a semantic segmentation model on it and testing on real-world automotive datasets. It specifically probes how dataset diversity (unique 3D assets) and training scale affect downstream perception performance.

## Datasets

- **VALERIE22** — total ?; splits: train (-1)
- **Cityscapes** — total ?; splits: test (-1)
- **A2D2** — total ?; splits: test (-1)
- **BDD100K** — total ?; splits: test (-1)
- **India Driving Dataset** — total ?; splits: test (-1)
- **Mapillary Vistas** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union computed over 11 shared semantic classes. Calculated as the average of (intersection of predicted and ground truth labels) / (union of predicted and ground truth labels) across all classes.

## Input / output format

**Input**: RGB images of urban street scenes.

**Output**: Pixel-wise semantic segmentation mask assigning one of 11 predefined classes (road, sidewalk, building, sky, car, truck, pole, traffic light, traffic sign, vegetation, person) to each pixel.

## Scoring recipe

```python
def compute_miou(pred_masks, gt_masks, num_classes=11):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        iou = intersection / union if union > 0 else 0.0
        ious.append(iou)
    return np.mean(ious)
```

## Common pitfalls

- Cross-domain evaluation requires strict 11-class mapping across synthetic and real datasets to ensure comparability.
- High frame count does not guarantee better generalization if asset diversity (unique 3D meshes) is low, leading to overfitting.
- Domain shift is significant when evaluating European-trained synthetic models on North American datasets like Mapillary Vistas.

## Evidence (verbatim from paper)

> To measure the performance of the task of semantic segmentation the mean Intersection over Union (mIoU) from the COCO semantic segmentation benchmark task is used [23]. The mIoU is denoted as the intersections between predicted semantic label classes and their corresponding ground truth divided by the union of the same, averaged over all classes.

## Citation

```bibtex
@misc{grau2023valerie22,
  title={VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments},
  author={Grau et al. (2023)},
  year={2023},
  note={arXiv:2308.09632}
}
```

- arXiv: 2308.09632