Monodepth Eval

---
name: monodepth-eval
description: Evaluates the accuracy and generalization of self-supervised monocular depth estimation models across image-based, pointcloud-based, and edge-based metrics on automotive and diverse natural scenes. Use when the user wants to benchmark on Kitti Eigen (KE split), Kitti Eigen-Benchmark (KEB split), SYNS-Patches, or asks about evaluating this task. Reports AbsRel, δ < 1.25^1, F-Score (pointcloud).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2208.01489
  bibtex_key: spencer2022deconstructing
  confidence: high
---

# monodepth-eval

> Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter — Spencer et al. (2022) (arXiv:2208.01489, 2022)

## What this evaluates

Evaluates the accuracy and generalization of self-supervised monocular depth estimation models across image-based, pointcloud-based, and edge-based metrics on automotive and diverse natural scenes.

## Datasets

- **Kitti Eigen (KE split)** — total ?; splits: test (-1)
- **Kitti Eigen-Benchmark (KEB split)** — total ?; splits: test (-1)
- **SYNS-Patches** — total 1175; splits: test (1175)

## Metrics

- `AbsRel` **(primary)** — range: other
  - Mean absolute relative error: (1/N) * sum(|pred - gt| / gt). Lower is better.
- `δ < 1.25^1` **(primary)** — range: percent
  - Percentage of pixels where max(pred/gt, gt/pred) < 1.25. Higher is better.
- `F-Score (pointcloud)` **(primary)** — range: percent
  - 3D pointcloud-based F-Score computed from reconstructed depth maps. Higher is better.
- `Chamfer` — range: other
  - Chamfer distance between predicted and ground truth pointclouds. Lower is better.
- `MAE` — range: other
  - Mean absolute error in meters. Lower is better.

## Input / output format

**Input**: Monocular video frames or stereo image pairs.

**Output**: Predicted depth map per input image.

## Scoring recipe

```python
1. Align predictions to ground truth: use per-image median scaling for monocular methods, fixed scaling factor for stereo methods.
2. Cap depth predictions at 100 meters.
3. Compute metrics (AbsRel, δ<1.25^1, F-Score, Chamfer, etc.) on aligned predictions vs. ground truth.
4. Average metrics over the dataset.
```

## Common pitfalls

- Using outdated ground truth (Eigen & Fergus 2015) instead of corrected depth maps (Uhrig et al. 2018) for KEB.
- Applying border cropping or stereo-blending post-processing, which the authors explicitly omit.
- Using a 50m or 80m depth cap instead of the specified 100m cap.
- Applying fixed scaling to monocular predictions instead of per-image median scaling.

## Evidence (verbatim from paper)

> We cap the maximum depth to 100 meters (compared to the common  $50\mathrm{m}$  (Garg et al., 2016) or  $80\mathrm{m}$  (Zhou et al., 2017)) and omit border cropping (Garg et al., 2016) & stereo-blending post-processing (Godard et al., 2017). Again, we show the rank ordering based on image-based (AbsRel), pointcloud-based (F-Score) and edge-based (F-Score) metrics. Monocular-supervised (M) methods instead apply per-image median scaling to align the prediction and ground-truth.

## Citation

```bibtex
@misc{spencer2022deconstructing,
  title={Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter},
  author={Spencer et al. (2022)},
  year={2022},
  note={arXiv:2208.01489}
}
```

- arXiv: 2208.01489