Per Object Depth Estimation Eval

---
name: per-object-depth-estimation-eval
description: Evaluates the accuracy of per-object depth estimation for vehicles in autonomous driving scenarios. It measures how well a model predicts the depth of individual objects given monocular RGB images and 2D bounding boxes. Use when the user wants to benchmark on Waymo Open Dataset, KITTI Detection Dataset, KITTI MOT Dataset, or asks about evaluating this task. Reports delta<1.25.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.03666
  bibtex_key: jing2022depthestimation
  confidence: high
---

# per-object-depth-estimation-eval

> Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking — Longlong Jing et al. (arXiv:2206.03666, 2022)

## What this evaluates

Evaluates the accuracy of per-object depth estimation for vehicles in autonomous driving scenarios. It measures how well a model predicts the depth of individual objects given monocular RGB images and 2D bounding boxes.

## Datasets

- **Waymo Open Dataset** — total 1000; splits: train (798), val (202)
- **KITTI Detection Dataset** — total 7480; splits: train (3712), test (3768)
- **KITTI MOT Dataset** — total 19103; splits: train (8008), test (11095)

## Metrics

- `delta<1.25` **(primary)** — range: percent
  - Percentage of object pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25. Higher is better.
- `Abs Rel` — range: other
  - Mean of |d_pred - d_gt| / d_gt across all object pixels. Lower is better.
- `Sq Rel` — range: other
  - Mean of (d_pred - d_gt)^2 / d_gt across all object pixels. Lower is better.
- `RMSE` — range: other
  - Root-mean-square error of depth predictions: sqrt(mean((d_pred - d_gt)^2)). Lower is better.
- `RMSE_log` — range: other
  - Root-mean-square error of log10 depth predictions: sqrt(mean((log10(d_pred) - log10(d_gt))^2)). Lower is better.

## Input / output format

**Input**: Monocular RGB image crop of the object, 2D bounding box coordinates, and optionally pseudo-LiDAR features or temporal tracklet frames.

**Output**: Per-object depth map or depth values for the object pixels.

## Scoring recipe

```python
def compute_depth_metrics(pred, gt):
    abs_rel = np.mean(np.abs(pred - gt) / gt)
    sq_rel = np.mean(((pred - gt) ** 2) / gt)
    rmse = np.sqrt(np.mean((pred - gt) ** 2))
    rmse_log = np.sqrt(np.mean((np.log10(pred) - np.log10(gt)) ** 2))
    delta = np.mean(np.maximum(pred / gt, gt / pred) < 1.25) * 100
    return {'Abs Rel': abs_rel, 'Sq Rel': sq_rel, 'RMSE': rmse, 'RMSE_log': rmse_log, 'delta<1.25': delta}
```

## Common pitfalls

- Evaluating with predicted 2D boxes instead of ground truth boxes, which conflates detection error with depth estimation error.
- Fusing temporal features without compensating for camera ego-motion, leading to misaligned depth predictions.
- Ignoring the vehicle-only focus, as the benchmark is specifically tuned for the vehicle class in autonomous driving.

## Evidence (verbatim from paper)

> Following the existing state-of-the-art per-object depth estimation benchmark proposed by [[27]], five standard metrics including average relative error (Abs Rel), squared relative error (Sq Rel), root-mean-square error (RMSE), average (log10) error (RMSE_log), and threshold accuracy (delta_i) are used for evaluation.

## Citation

```bibtex
@misc{jing2022depthestimation,
  title={Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking},
  author={Longlong Jing et al.},
  year={2022},
  note={arXiv:2206.03666}
}
```

- arXiv: 2206.03666