E3d Bench Eval

---
name: e3d-bench-eval
description: Evaluates the effectiveness, robustness, and inference efficiency of end-to-end 3D Geometric Foundation Models across sparse-view depth estimation, video depth estimation, and multi-view relative pose estimation. It probes models' ability to generalize across diverse domains including indoor, outdoor, aerial, and highly dynamic scenes under both normalized and metric-scale settings. Use when the user wants to benchmark on DTU, ETH3D, KITTI, Tanks and Temples, ScanNet, Bonn, TUM Dynamics, Sintel, PointOdyssey, Syndrome, CO3Dv2, RealEstate10K, ScanNet-eval, KITTI Odometry, ADT, ACID, ULTRRA, or asks about evaluating this task. Reports AbsRel.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01933
  bibtex_key: cong2025e3dbench
  confidence: high
---

# e3d-bench-eval

> E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models — Wenyan Cong et al. (2025) (arXiv:2506.01933, 2025)

## What this evaluates

Evaluates the effectiveness, robustness, and inference efficiency of end-to-end 3D Geometric Foundation Models across sparse-view depth estimation, video depth estimation, and multi-view relative pose estimation. It probes models' ability to generalize across diverse domains including indoor, outdoor, aerial, and highly dynamic scenes under both normalized and metric-scale settings.

## Datasets

- **DTU** — total ?; splits: test (-1)
- **ETH3D** — total ?; splits: test (-1)
- **KITTI** — total ?; splits: test (-1)
- **Tanks and Temples** — total ?; splits: test (-1)
- **ScanNet** — total ?; splits: test (-1)
- **Bonn** — total ?; splits: test (-1)
- **TUM Dynamics** — total ?; splits: test (-1)
- **Sintel** — total ?; splits: test (-1)
- **PointOdyssey** — total ?; splits: val (-1)
- **Syndrome** — total ?; splits: test (-1)
- **CO3Dv2** — total ?; splits: test (-1)
- **RealEstate10K** — total ?; splits: test (-1)
- **ScanNet-eval** — total ?; splits: test (-1)
- **KITTI Odometry** — total ?; splits: test (-1)
- **ADT** — total ?; splits: test (-1)
- **ACID** — total ?; splits: test (-1)
- **ULTRRA** — total ?; splits: test (-1)

## Metrics

- `AbsRel` **(primary)** — range: other
  - Measures the average relative error between model output and ground truth: mean(|pred - gt| / gt). Lower is better.
- `δ < 1.03` — range: percent
  - Captures the percentage of pixels within 3% relative error (max(pred/gt, gt/pred) < 1.03). Higher is better.
- `δ < 1.25` — range: percent
  - Captures the percentage of pixels within 25% relative error (max(pred/gt, gt/pred) < 1.25). Higher is better.
- `ATE` — range: other
  - Absolute Translation Error between predicted and ground-truth trajectories after Sim(3) alignment. Lower is better.
- `RPE-trans` — range: other
  - Relative Translation Error between consecutive predicted and ground-truth poses after Sim(3) alignment. Lower is better.
- `RPE-rot` — range: other
  - Relative Rotation Error between consecutive predicted and ground-truth poses after Sim(3) alignment. Lower is better.

## Input / output format

**Input**: Sparse-view image pairs or triplets with minimal/no overlap for depth estimation; monocular video sequences for video depth estimation; multi-view image collections for relative pose estimation.

**Output**: Per-pixel depth maps (extracted from the z-coordinate of predicted point maps or via confidence-weighted averaging); relative camera pose trajectories (translation and rotation matrices).

## Scoring recipe

```python
def compute_depth_metrics(pred_depth, gt_depth, delta_threshold=1.03):
    abs_rel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
    delta = np.mean(np.maximum(pred_depth / gt_depth, gt_depth / pred_depth) < delta_threshold)
    return abs_rel, delta

def compute_pose_metrics(pred_traj, gt_traj):
    pred_aligned = align_sim3(pred_traj, gt_traj)
    ate = np.mean(np.linalg.norm(pred_aligned - gt_traj, axis=-1))
    rpe_trans = np.mean(np.linalg.norm(np.diff(pred_aligned, axis=0) - np.diff(gt_traj, axis=0), axis=-1))
    rpe_rot = np.mean(np.arccos(np.clip(np.einsum('ij,ij->i', np.diff(pred_aligned, axis=0), np.diff(gt_traj, axis=0)), -1, 1)))
    return ate, rpe_trans, rpe_rot
```

## Common pitfalls

- Confusing normalized vs. metric-scale evaluation protocols; normalized models require median depth scaling, while metric models should be evaluated on raw outputs or with an extra median-aligned setting.
- Failing to upsample all predictions to full resolution before computing metrics, which inflates error values.
- Not handling video padding or frame repetition for models that require fixed input lengths (e.g., padding sequences <16 frames for Geo4D, or using default 41-frame outputs for Aether).
- Applying a single Sim(3) alignment to datasets with separate coordinate systems (e.g., ULTRRA aerial/ground trajectories), which requires alternative metrics.

## Evidence (verbatim from paper)

> Evaluation Metrics: Two widely used metrics to evaluate depth estimation performance: Absolute Relative Error (AbsRel) measures the average relative error between model output and ground truth; the Inlier Ratio δ < 1.03 [36, 37] captures the percentage of pixels within 3% relative error.

## Citation

```bibtex
@misc{cong2025e3dbench,
  title={E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models},
  author={Wenyan Cong et al. (2025)},
  year={2025},
  note={arXiv:2506.01933}
}
```

- arXiv: 2506.01933