Mai Aim2022 Depth Eval

---
name: mai-aim2022-depth-eval
description: Evaluates the accuracy and inference speed of monocular depth estimation models on resource-constrained mobile devices. It measures depth prediction quality using invariant standard root mean squared error and records inference time on a Raspberry Pi 4 to assess real-time capability. Use when the user wants to benchmark on MAI&AIM2022 challenge dataset, or asks about evaluating this task. Reports si-RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.00961
  bibtex_key: li2022litedepth
  confidence: high
---

# mai-aim2022-depth-eval

> LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devices — Li et al. (2022) (arXiv:2209.00961, 2022)

## What this evaluates

Evaluates the accuracy and inference speed of monocular depth estimation models on resource-constrained mobile devices. It measures depth prediction quality using invariant standard root mean squared error and records inference time on a Raspberry Pi 4 to assess real-time capability.

## Datasets

- **MAI&AIM2022 challenge dataset** — total 7385; splits: train (6869), val (516)

## Metrics

- `si-RMSE` **(primary)** — range: other
  - Invariant standard root mean squared error. Computes the RMSE of the log-depth difference after removing the mean shift between prediction and ground truth, making it scale- and shift-invariant.
- `runtime` — range: other
  - Inference time measured in milliseconds on a Raspberry Pi 4 using TFLite.
- `Score` — range: other
  - Score(si-RMSE, runtime) = (2^{-20} * si-RMSE) / (C * runtime), where C=0.01 on the online validation benchmark. Higher values indicate better combined accuracy and speed.

## Input / output format

**Input**: RGB image (640x480 resolution, float32)

**Output**: Predicted depth map (640x480 resolution, uint16 format representing 0-40 meters)

## Scoring recipe

```python
def compute_si_rmse(pred, gt):
    log_pred = np.log(pred)
    log_gt = np.log(gt)
    diff = log_pred - log_gt - np.mean(log_pred - log_gt)
    return np.sqrt(np.mean(diff**2))

def compute_score(si_rmse, runtime_ms):
    C = 0.01
    return (2**-20 * si_rmse) / (C * runtime_ms)
```

## Common pitfalls

- Runtime must be measured on a Raspberry Pi 4 using TFLite, not on desktop GPUs or other hardware.
- Depth maps are stored as uint16 values representing 0-40 meters; incorrect scaling or type conversion will break the metric.
- The scoring formula uses a constant C=0.01 specifically for the online validation benchmark, which directly impacts the final ranking.

## Evidence (verbatim from paper)

> In MAI&AIM2022 challenge, two metrics are considered for each submission solution: 1) The quality of the depth estimation. It is measured by the invariant standard root mean squared error (si-RMSE). 2) The runtime of the model on the target platform (i.e., Raspberry Pi 4). The scoring formulation is provided below: Score(si-RMSE, runtime) = (2^{-20} * si-RMSE) / (C * runtime), where C=0.01 on the online validation benchmark.

## Citation

```bibtex
@misc{li2022litedepth,
  title={LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devices},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2209.00961}
}
```

- arXiv: 2209.00961