Co Semdepth Eval

---
name: co-semdepth-eval
description: Evaluates a joint deep learning architecture for simultaneous monocular depth estimation and semantic segmentation on aerial drone imagery, measuring prediction accuracy and inference speed against single-task and joint baselines. Use when the user wants to benchmark on MidAir, Aeroscapes, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.17982
  bibtex_key: alaaeldin2025cosemdepth
  confidence: high
---

# co-semdepth-eval

> Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on Aerial Images — AlaaEldin et al. (2025) (arXiv:2503.17982, 2025)

## What this evaluates

Evaluates a joint deep learning architecture for simultaneous monocular depth estimation and semantic segmentation on aerial drone imagery, measuring prediction accuracy and inference speed against single-task and joint baselines.

## Datasets

- **MidAir** — total 420000; splits: train (-1), val (-1), test (-1)
- **Aeroscapes** — total 3269; splits: train (-1), test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union across all semantic classes. Computed as the average of per-class IoU (intersection over union of predicted and ground truth masks).
- `RMSE` — range: other
  - Linear root mean square error between predicted and ground truth depth maps.
- `AbsRelErr` — range: other
  - Absolute relative error, computed as the mean of |predicted - ground truth| / ground truth over valid pixels.
- `δ<1.25` — range: percent
  - Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25.
- `δ<1.25^2` — range: percent
  - Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25^2.
- `δ<1.25^3` — range: percent
  - Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25^3.
- `Inference Time` — range: other
  - Average time to process one frame, measured in milliseconds per frame (ms/f).

## Input / output format

**Input**: Monocular RGB images (384x384 for MidAir, 1280x720 for Aeroscapes). For joint evaluation, video sequences are used to leverage temporal information.

**Output**: Predicted depth maps and semantic segmentation maps. Depth and semantic maps are initially predicted at half the input resolution and then upsampled to original resolution using nearest-neighbor interpolation.

## Scoring recipe

```python
# Depth metrics
valid_pixels = ground_truth_depth > 0
pred_depth = pred_depth[valid_pixels]
gt_depth = gt_depth[valid_pixels]
rmse = np.sqrt(np.mean((pred_depth - gt_depth)**2))
abs_rel_err = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
ratios = np.maximum(pred_depth / gt_depth, gt_depth / pred_depth)
delta1 = np.mean(ratios < 1.25) * 100
delta2 = np.mean(ratios < 1.25**2) * 100
delta3 = np.mean(ratios < 1.25**3) * 100

# Semantic metric
per_class_iou = []
for class_id in range(num_classes):
    pred_mask = (pred_semantic == class_id)
    gt_mask = (gt_semantic == class_id)
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0
    per_class_iou.append(iou)
miou = np.mean(per_class_iou)
```

## Common pitfalls

- Depth values are capped at 80.0 meters during evaluation, which truncates long-range predictions.
- MidAir's original 14 semantic classes are mapped to 7 classes (e.g., Ground Vegetation, Rocky Ground, Dirt Ground → Land) before evaluation.
- Aeroscapes lacks depth annotations, so it is only used for evaluating the single-task semantic segmentation baseline (M4Semantic), not the joint architecture.
- Inference is performed at half resolution and upsampled via nearest-neighbor, which may slightly reduce accuracy but is required for memory constraints.

## Evidence (verbatim from paper)

> To quantitatively evaluate the depth prediction results, we consider the commonly used evaluation metrics in prior works[[11], [3], [16]]. These include the linear root mean square error (RMSE), the absolute relative error, and accuracy under a threshold. For semantic segmentation, we use the commonly used mean Intersection over Union $mIoU$ metric. The Inference Time (Inf. Time) is computed in milliseconds per frame (ms/f).

## Citation

```bibtex
@misc{alaaeldin2025cosemdepth,
  title={Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on Aerial Images},
  author={AlaaEldin et al. (2025)},
  year={2025},
  note={arXiv:2503.17982}
}
```

- arXiv: 2503.17982