Mdlbench Eval

---
name: mdlbench-eval
description: Evaluates the inference performance and overhead of six major mobile deep learning libraries across diverse model architectures, tasks, and mobile hardware configurations. It measures how software-level optimizations and library choices impact on-device latency compared to hardware capabilities and algorithmic optimizations like quantization. Use when the user wants to benchmark on MDLBench, or asks about evaluating this task. Reports inference time.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.06512
  bibtex_key: zhang2022mdlbench
  confidence: high
---

# mdlbench-eval

> Benchmarking of DL Libraries and Models on Mobile Devices — Zhang et al. (2022) (arXiv:2202.06512, 2022)

## What this evaluates

Evaluates the inference performance and overhead of six major mobile deep learning libraries across diverse model architectures, tasks, and mobile hardware configurations. It measures how software-level optimizations and library choices impact on-device latency compared to hardware capabilities and algorithmic optimizations like quantization.

## Datasets

- **MDLBench** — total 15; splits: test (15)

## Metrics

- `inference time` **(primary)** — range: other
  - Average latency measured over N iterations (default N=50) after a warm-up phase. Includes per-operator latency and duration traces collected via library instrumentation or adb.

## Input / output format

**Input**: Pre-trained deep learning models (e.g., MobileNet, ResNet, YOLO, ALBERT) converted to library-specific formats, executed on mobile devices with specified hardware configurations (CPU/GPU/DSP) and precision levels (float32/int8).

**Output**: Inference latency (time per inference), per-operator latency/duration traces, and input/output dimensions, written to device storage and retrieved to desktop.

## Scoring recipe

```python
def compute_avg_latency(model, lib, device, iterations=50):
    # 1. Push lib and model to device via adb
    # 2. Kill background apps, set CPU freq, use 4 big cores
    # 3. Load lib and model into memory (warm-up)
    latencies = []
    for _ in range(iterations):
        t_start = current_time()
        lib.run(model)
        t_end = current_time()
        latencies.append(t_end - t_start)
    # 4. Retrieve results from device storage
    return sum(latencies) / len(latencies)
```

## Common pitfalls

- Performance varies significantly across hardware accelerators (CPU/GPU/DSP) and precision (float32 vs int8); results are not directly comparable without specifying the exact configuration.
- The benchmark measures raw inference latency, not model accuracy or task performance, so it does not evaluate the correctness of the models.
- Background processes and system thermal throttling can affect results; the protocol explicitly requires killing background apps and controlling CPU frequency.

## Evidence (verbatim from paper)

> MDLBench profiles the inference time and operator-level information, e.g., per-operator latency, duration, input/output dimension, etc.

## Citation

```bibtex
@misc{zhang2022mdlbench,
  title={Benchmarking of DL Libraries and Models on Mobile Devices},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2202.06512}
}
```

- arXiv: 2202.06512