Mdec Syns Patches Eval

---
name: mdec-syns-patches-eval
description: Evaluates monocular depth estimation models across diverse real-world environments (natural, agricultural, urban, indoor) using high-quality LiDAR ground truth. Probes zero-shot generalization, boundary interpolation accuracy, and robustness to scene diversity and image artifacts. Use when the user wants to benchmark on SYNS-Patches, or asks about evaluating this task. Reports F-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.07051
  bibtex_key: spencer2023mdec
  confidence: high
---

# mdec-syns-patches-eval

> The Second Monocular Depth Estimation Challenge — Spencer et al. (2023) (arXiv:2304.07051, 2023)

## What this evaluates

Evaluates monocular depth estimation models across diverse real-world environments (natural, agricultural, urban, indoor) using high-quality LiDAR ground truth. Probes zero-shot generalization, boundary interpolation accuracy, and robustness to scene diversity and image artifacts.

## Datasets

- **SYNS-Patches** — total ?; splits: test (-1)

## Metrics

- `F-Score` **(primary)** — range: percent
  - Harmonic mean of precision and recall for depth estimation, typically computed at a specific threshold or for edge detection. Used as the primary ranking metric.
- `F-Edges` — range: percent
  - F-Score computed specifically on depth boundaries/edges to evaluate contour accuracy.
- `MAE` — range: other
  - Mean Absolute Error between predicted and ground truth depth values.
- `RMSE` — range: other
  - Root Mean Squared Error between predicted and ground truth depth values.
- `AbsRel` — range: other
  - Mean absolute relative error, calculated as the mean of |pred - gt| / gt across valid pixels.
- `Acc-Edges` — range: percent
  - Edge accuracy metric measuring the proportion of correctly predicted edge pixels.
- `Comp-Edges` — range: percent
  - Edge completion metric measuring the proportion of ground truth edges successfully recovered by the prediction.

## Input / output format

**Input**: Monocular RGB image

**Output**: Predicted depth map (single-channel float tensor)

## Scoring recipe

```python
def compute_metrics(pred, gt):
    # Median alignment for fair comparison across supervision types
    pred_aligned = pred * (gt.median() / pred.median())
    
    # F-Score & F-Edges (edge-based)
    pred_edges = detect_edges(pred_aligned)
    gt_edges = detect_edges(gt)
    tp = (pred_edges & gt_edges).sum()
    precision = tp / pred_edges.sum() if pred_edges.sum() > 0 else 0
    recall = tp / gt_edges.sum() if gt_edges.sum() > 0 else 0
    f_score = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    
    # MAE, RMSE, AbsRel
    mae = np.mean(np.abs(pred_aligned - gt))
    rmse = np.sqrt(np.mean((pred_aligned - gt)**2))
    absrel = np.mean(np.abs(pred_aligned - gt) / gt)
    
    return f_score, f_edges, mae, rmse, absrel
```

## Common pitfalls

- Models must be median-aligned to the ground truth before evaluation to ensure fair comparisons across different supervision types.
- Self-supervised methods trained exclusively on automotive data fail to generalize to indoor or non-automotive scenes.
- Thin structures (e.g., railings, branches) and transparent surfaces (e.g., glass) are consistently poorly estimated due to interpolation halos and lack of LiDAR/photometric constraints.

## Evidence (verbatim from paper)

> Participant submissions were evaluated on SYNS-Patches [1, 78]. As previously mentioned, this paper only discusses submissions that outperformed the baseline in any pointcloud-/image-based metric across the Overall dataset. Since both challenge phases ran independently and participants were responsible for generating the predictions, we cannot guarantee that the testing/Validation metrics used the same model. We therefore report results only for the test split. All methods were median aligned w.r.t. the ground-truth, regardless of the supervision used. This ensures that the evaluations are identical and comparisons are fair. Table 2 shows the overall performance for each submission across the whole dataset, as well as each category. Each subset is ordered using F-Score performance.

## Citation

```bibtex
@misc{spencer2023mdec,
  title={The Second Monocular Depth Estimation Challenge},
  author={Spencer et al. (2023)},
  year={2023},
  note={arXiv:2304.07051}
}
```

- arXiv: 2304.07051