Mdec Synspatches Eval

---
name: mdec-synspatches-eval
description: Evaluates monocular depth estimation models on their ability to predict accurate depth maps from single images. It specifically probes zero-shot generalization across diverse natural and indoor scenes using LiDAR-ground truth. Use when the user wants to benchmark on SYNS-Patches, or asks about evaluating this task. Reports F-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.17787
  bibtex_key: obukhov2025mdec
  confidence: high
---

# mdec-synspatches-eval

> The Fourth Monocular Depth Estimation Challenge — Anton Obukhov et al. (2025) (arXiv:2504.17787, 2025)

## What this evaluates

Evaluates monocular depth estimation models on their ability to predict accurate depth maps from single images. It specifically probes zero-shot generalization across diverse natural and indoor scenes using LiDAR-ground truth.

## Datasets

- **SYNS-Patches** — total ?; splits: test (-1); repo https://github.com/toshas/mdec_benchmark

## Metrics

- `F-Score` **(primary)** — range: percent
  - Harmonic mean of precision and recall for depth predictions, computed after aligning predictions to ground truth via least-squares or median scaling. Used as the primary ranking metric.
- `AbsRel` — range: percent
  - Absolute relative error between predicted and ground truth depth, averaged over valid pixels.
- `MAE` — range: other
  - Mean absolute error between predicted and ground truth depth.
- `RMSE` — range: other
  - Root mean squared error between predicted and ground truth depth.
- `Acc-Edges` — range: percent
  - Accuracy of predicted depth edges compared to ground truth edges.
- `F-Edges` — range: percent
  - F-Score computed specifically on depth edges.
- `δ<1.25` — range: percent
  - Percentage of pixels where predicted depth is within a factor of 1.25 of ground truth.
- `δ<1.25^3` — range: percent
  - Percentage of pixels where predicted depth is within a factor of 1.25^3 of ground truth.

## Input / output format

**Input**: Single RGB image

**Output**: Predicted depth map (evaluated after alignment to ground truth)

## Scoring recipe

```python
def evaluate(pred_depth, gt_depth, align='least_squares'):
    if align == 'median':
        scale = np.median(gt_depth) / np.median(pred_depth)
        shift = 0.0
    else:
        # least-squares alignment with two degrees of freedom
        scale, shift = np.linalg.lstsq(...)
    pred_aligned = pred_depth * scale + shift
    f_score = compute_f1(pred_aligned, gt_depth)
    absrel = np.mean(np.abs(pred_aligned - gt_depth) / gt_depth)
    mae = np.mean(np.abs(pred_aligned - gt_depth))
    rmse = np.sqrt(np.mean((pred_aligned - gt_depth)**2))
    return {'F-Score': f_score, 'AbsRel': absrel, 'MAE': mae, 'RMSE': rmse}
```

## Common pitfalls

- Forgetting to align predictions to ground truth before computing metrics; the protocol explicitly requires least-squares or median scaling.
- Confusing the primary ranking metric (F-Score) with image-based metrics like AbsRel or δ<1.25, which may rank teams differently.
- Assuming the benchmark uses metric depth; it explicitly uses affine-invariant/disparity-invariant evaluation.

## Evidence (verbatim from paper)

> Following the protocol from previous editions, submitted predictions were evaluated on the testing split of SYNS-Patches *[[1], [96]]*, after being aligned to ground-truth depths according to median depth scaling or least-squares alignment, as requested by the participants. TableLABEL:tbl:res:results collects the results of this fourth edition of the challenge, ranking the submitted methods according to their F-Score performance.

## Citation

```bibtex
@misc{obukhov2025mdec,
  title={The Fourth Monocular Depth Estimation Challenge},
  author={Anton Obukhov et al. (2025)},
  year={2025},
  note={arXiv:2504.17787}
}
```

- arXiv: 2504.17787