Mtcityscapes 3d Eval

---
name: mtcityscapes-3d-eval
description: Evaluates joint 2D-3D multi-task scene understanding on urban street imagery. It probes a model's ability to concurrently perform monocular 3D vehicle detection, 19-class semantic segmentation, and monocular depth estimation. Use when the user wants to benchmark on MTCityscapes-3D, or asks about evaluating this task. Reports mDS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.00971
  bibtex_key: ye2023joint2d3d
  confidence: high
---

# mtcityscapes-3d-eval

> Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimation — Ye et al. (2023) (arXiv:2304.00971, 2023)

## What this evaluates

Evaluates joint 2D-3D multi-task scene understanding on urban street imagery. It probes a model's ability to concurrently perform monocular 3D vehicle detection, 19-class semantic segmentation, and monocular depth estimation.

## Datasets

- **MTCityscapes-3D** — total 3475; splits: train (2975), val (500)

## Metrics

- `mDS` **(primary)** — range: [0, 1]
  - Mean detection score computed using the official Cityscapes-3D evaluation script.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union averaged across 19 semantic classes.
- `RMSE` — range: other
  - Root Mean Square Error between predicted and ground-truth depth values.

## Input / output format

**Input**: RGB images at original resolution 1024×2048.

**Output**: Per image: 3D bounding boxes (location, rotation, dimensions, class, center-ness, direction) for vehicles; 19-class semantic segmentation map; monocular depth map.

## Scoring recipe

```python
def evaluate(preds, gold):
    # 3D Detection
    mDS = cityscapes3d_official_eval(preds['3d_boxes'], gold['3d_boxes'])
    # Segmentation
    ious = [intersection_over_union(preds['seg'][c], gold['seg'][c]) for c in range(19)]
    mIoU = sum(ious) / 19
    # Depth
    RMSE = sqrt(mean((preds['depth'] - gold['depth'])**2))
    return {'mDS': mDS, 'mIoU': mIoU, 'RMSE': RMSE}
```

## Common pitfalls

- Evaluation is strictly performed at the original 1024×2048 resolution, despite training occurring at 768×1536.
- The 3D detection metric (mDS) relies on the official Cityscapes-3D script, which differs from standard KITTI/nuScenes protocols.
- Depth ground truth is derived from stereo cameras rather than LiDAR, introducing potential modality discrepancies.

## Evidence (verbatim from paper)

> The 3D vehicle detection task (3Ddet) utilizes the mean detection score (mDS) as its metric, using the official evaluation script provided by Cityscapes-3D. The models are assessed on the validation set for all tasks.

## Citation

```bibtex
@misc{ye2023joint2d3d,
  title={Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimation},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2304.00971}
}
```

- arXiv: 2304.00971