Robodepth Eval

---
name: robodepth-eval
description: This benchmark evaluates the out-of-distribution robustness of monocular depth estimation (MDE) models when exposed to real-world corruptions such as weather changes, sensor failures, and data processing anomalies. It measures how much model performance degrades relative to a clean baseline across multiple corruption types and severity levels. Use when the user wants to benchmark on KITTI-C, NYUDepth2-C, KITTI-S, or asks about evaluating this task. Reports mCE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.15171
  bibtex_key: kong2023robodepth
  confidence: high
---

# robodepth-eval

> RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions — Kong et al. (2023) (arXiv:2310.15171, 2023)

## What this evaluates

This benchmark evaluates the out-of-distribution robustness of monocular depth estimation (MDE) models when exposed to real-world corruptions such as weather changes, sensor failures, and data processing anomalies. It measures how much model performance degrades relative to a clean baseline across multiple corruption types and severity levels.

## Datasets

- **KITTI-C** — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md
- **NYUDepth2-C** — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md
- **KITTI-S** — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md

## Metrics

- `mCE` **(primary)** — range: percent
  - Mean Corruption Error. Reported as a percentage (%) representing the average relative depth error degradation compared to a clean baseline model across all corruption types.
- `mRR` — range: percent
  - Mean Relative Robustness. Reported as a percentage (%) measuring the average relative robustness score compared to a clean baseline across all corruption types.
- `mDEE` — range: other
  - Mean Depth Estimation Error. The absolute depth estimation error averaged across the dataset.

## Input / output format

**Input**: Corrupted images (monocular, stereo, or monocular+stereo pairs) from outdoor or indoor scenes.

**Output**: Predicted depth maps for each input image.

## Scoring recipe

```python
# For each model and corruption type:
error_clean = compute_depth_error(model, clean_images)
error_corrupted = compute_depth_error(model, corrupted_images)

# Calculate relative degradation
relative_errors = [error_corrupted[c] / error_clean[c] for c in corruption_types]
mCE = mean(relative_errors) * 100

# Calculate relative robustness
relative_robustness = [1 - (error_corrupted[c] - error_clean[c]) / error_clean[c] for c in corruption_types]
mRR = mean(relative_robustness) * 100

# Absolute error
mDEE = mean(error_corrupted)
```

## Common pitfalls

- Failing to unify model configurations (backbones, data augmentations, post-processing) across candidates leads to unfair comparisons.
- Confusing mCE/mRR (relative degradation percentages) with raw absolute error values.
- Evaluating only on clean validation splits instead of the corrupted KITTI-C/NYUDepth2-C datasets misses the benchmark's core objective.

## Evidence (verbatim from paper)

> The mCE and mRR scores are given in percentage (%). Blocks from top to bottom: [1st]: The baseline MonoDepth2; [2nd]: Methods w/ monocular inputs; [3rd]: Methods w/ stereo inputs; [4th]: Methods w/ monocular + stereo inputs.

## Citation

```bibtex
@misc{kong2023robodepth,
  title={RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions},
  author={Kong et al. (2023)},
  year={2023},
  note={arXiv:2310.15171}
}
```

- arXiv: 2310.15171