Kitti Depth Eval

---
name: kitti-depth-eval
description: Evaluates self-supervised monocular depth estimation models on outdoor driving scenes, measuring both the geometric accuracy of predicted depth maps and the reliability of associated uncertainty estimates. Use when the user wants to benchmark on KITTI, or asks about evaluating this task. Reports Abs Rel.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.09692
  bibtex_key: zhou2021subdepth
  confidence: high
---

# kitti-depth-eval

> SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation — Zhou et al. (2021) (arXiv:2111.09692, 2021)

## What this evaluates

Evaluates self-supervised monocular depth estimation models on outdoor driving scenes, measuring both the geometric accuracy of predicted depth maps and the reliability of associated uncertainty estimates.

## Datasets

- **KITTI** — total ?; splits: train (39810), val (4424)

## Metrics

- `Abs Rel` **(primary)** — range: [0, 1]
  - Mean absolute relative error: (1/N) Σ |d_pred - d_gt| / d_gt. Lower is better.
- `Sq Rel` — range: [0, 1]
  - Mean squared relative error: (1/N) Σ (d_pred - d_gt)² / d_gt. Lower is better.
- `RMSE` — range: [0, inf)
  - Root mean squared error: sqrt((1/N) Σ (d_pred - d_gt)²). Lower is better.
- `RMSE log` — range: [0, inf)
  - Root mean squared error of log-depth: sqrt((1/N) Σ (log d_pred - log d_gt)²). Lower is better.
- `δ1` — range: [0, 1]
  - Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25). Higher is better.
- `δ2` — range: [0, 1]
  - Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25²). Higher is better.
- `δ3` — range: [0, 1]
  - Accuracy threshold: (1/N) Σ I(max(d_pred/d_gt, d_gt/d_pred) < 1.25³). Higher is better.
- `AUSE` — range: [0, 1]
  - Area Under the Sparsification Error curve. Measures how quickly uncertainty estimates correlate with actual error. Lower is better.
- `AURG` — range: [0, 1]
  - Area Under the Random Gain curve. Measures uncertainty calibration quality. Higher is better.

## Input / output format

**Input**: RGB image frames resized to 640×192. Monocular frame triplets with a universal camera intrinsic matrix.

**Output**: Predicted dense depth map and per-pixel depth uncertainty map.

## Scoring recipe

```python
def compute_depth_metrics(pred, gt):
    valid = gt > 0
    p, g = pred[valid], gt[valid]
    abs_rel = np.mean(np.abs(p - g) / g)
    sq_rel = np.mean(((p - g) ** 2) / g)
    rmse = np.sqrt(np.mean((p - g) ** 2))
    rmse_log = np.sqrt(np.mean((np.log(p) - np.log(g)) ** 2))
    thresh = np.maximum((p / g), (g / p))
    d1 = np.mean(thresh < 1.25)
    d2 = np.mean(thresh < 1.25 ** 2)
    d3 = np.mean(thresh < 1.25 ** 3)
    return abs_rel, sq_rel, rmse, rmse_log, d1, d2, d3
```

## Common pitfalls

- Confusing the Eigen split with other KITTI splits (e.g., Geiger) changes train/val/test sizes.
- AUSE and AURG have opposite optimization directions: lower AUSE is better, higher AURG is better.
- Photometric loss weighting is often confused with uncertainty weighting in ablation studies; the paper explicitly compares 1:1 fixed weighting vs. learned uncertainty weighting.

## Evidence (verbatim from paper)

> Depth metrics described by Eigen [9] are the most common used metrics for evaluating depth estimation accuracy. They include four error metrics: the Absolute Relative Error (Abs Rel), Squared Relative Error (Sq Rel), Root Mean Squared Error (RMSE), and the log of RMSE; accuracy metric: δ1, δ2, δ3.

## Citation

```bibtex
@misc{zhou2021subdepth,
  title={SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation},
  author={Zhou et al. (2021)},
  year={2021},
  note={arXiv:2111.09692}
}
```

- arXiv: 2111.09692