Kitti Eigen Depth Eval

---
name: kitti-eigen-depth-eval
description: Evaluates the accuracy of self-supervised monocular depth estimation models on urban driving scenes. It probes the model's ability to predict per-pixel depth from a single image or video sequence, handling occlusions, moving objects, and scale ambiguity. Use when the user wants to benchmark on KITTI 2015 (Eigen split), Make3D, or asks about evaluating this task. Reports Abs Rel, δ < 1.25.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.01260
  bibtex_key: godard2018digging
  confidence: high
---

# kitti-eigen-depth-eval

> Digging Into Self-Supervised Monocular Depth Estimation — Godard et al. (2018) (arXiv:1806.01260, 2018)

## What this evaluates

Evaluates the accuracy of self-supervised monocular depth estimation models on urban driving scenes. It probes the model's ability to predict per-pixel depth from a single image or video sequence, handling occlusions, moving objects, and scale ambiguity.

## Datasets

- **KITTI 2015 (Eigen split)** — total ?; splits: train (39810), val (4424), test (-1)
- **Make3D** — total ?; splits: test (-1)

## Metrics

- `Abs Rel` **(primary)** — range: other
  - Mean absolute relative error: average of |d_pred - d_gt| / d_gt across valid pixels.
- `δ < 1.25` **(primary)** — range: [0, 1]
  - Fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25.
- `RMSE` — range: other
  - Root mean squared error: sqrt(mean((d_pred - d_gt)^2)) across valid pixels.

## Input / output format

**Input**: Single RGB image (or sequence of frames for video models) typically at 512×160 or 1024×320 resolution.

**Output**: Per-pixel depth map (H×W matrix of float values representing distance in meters).

## Scoring recipe

```python
def compute_metrics(pred_depth, gt_depth):
    # Apply per-image median scaling for monocular models
    scale = np.median(gt_depth) / np.median(pred_depth)
    pred_scaled = pred_depth * scale
    # Filter invalid pixels (depth > 80m or <= 0)
    valid = (gt_depth > 0) & (gt_depth <= 80) & (pred_scaled > 0)
    d_pred = pred_scaled[valid]
    d_gt = gt_depth[valid]
    abs_rel = np.mean(np.abs(d_pred - d_gt) / d_gt)
    rmse = np.sqrt(np.mean((d_pred - d_gt)**2))
    delta = np.mean(np.maximum(d_pred/d_gt, d_gt/d_pred) < 1.25)
    return abs_rel, rmse, delta
```

## Common pitfalls

- Per-image median scaling must be applied for monocular models but NOT for stereo-supervised models, as scale is recovered from the known baseline during training.
- Results are reported without post-processing; applying standard post-processing significantly improves scores but is not the default reported protocol.
- The model assumes Lambertian surfaces and fails on moving objects or non-static scenes, which requires auto-masking during training but isn't handled during evaluation.

## Evidence (verbatim from paper)

> We evaluate our models, named Monodepth2, on the KITTI 2015 stereo dataset [13], to allow comparison with previously published monocular methods. For our monocular models, we report results using the per-image median ground truth scaling introduced by [76]. See also supplementary material Section D.2 for results where we apply a single median scaling to the whole test set, instead of scaling each image independently. Table 1. Quantitative results. Comparison of our method to existing methods on KITTI 2015 [13] using the Eigen split. ... Abs Rel ... Sq Rel ... RMSE ... RMSE log ... δ < 1.25 ...

## Citation

```bibtex
@misc{godard2018digging,
  title={Digging Into Self-Supervised Monocular Depth Estimation},
  author={Godard et al. (2018)},
  year={2018},
  note={arXiv:1806.01260}
}
```

- arXiv: 1806.01260