Mobile Inference Benchmark Eval

---
name: mobile-inference-benchmark-eval
description: Evaluates the practical feasibility of running deep learning models on mobile hardware by measuring end-to-end latency and energy consumption during CNN inference. It compares on-device execution against cloud-based execution to identify hardware and network bottlenecks. Use when the user wants to benchmark on Mobile Benchmark Image Set, or asks about evaluating this task. Reports end-to-end latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1707.04610
  bibtex_key: guo2017mobileinference
  confidence: medium
---

# mobile-inference-benchmark-eval

> Cloud-based or On-device: An Empirical Study of Mobile Deep Inference — Guo (2017) (arXiv:1707.04610, 2017)

## What this evaluates

Evaluates the practical feasibility of running deep learning models on mobile hardware by measuring end-to-end latency and energy consumption during CNN inference. It compares on-device execution against cloud-based execution to identify hardware and network bottlenecks.

## Datasets

- **Mobile Benchmark Image Set** — total 15; splits: test (15)

## Metrics

- `end-to-end latency` **(primary)** — range: other
  - Measures the total time from image input to receiving the top-5 label output, including model loading, computation, and network transfer (for cloud mode).
- `energy consumption` — range: other
  - Total power draw measured during the inference process, capturing overhead from model loading and computation.

## Input / output format

**Input**: 224x224 pixel image bitmaps (original or downscaled).

**Output**: Probability distribution over labels, specifically the top 5 most probable labels.

## Scoring recipe

```python
for each image in test_set:
  start_t = time.perf_counter()
  start_e = measure_power()
  output = run_inference(image) # on-device or cloud
  end_t = time.perf_counter()
  end_e = measure_power()
  latencies.append(end_t - start_t)
  energies.append(end_e - start_e)
avg_latency = sum(latencies) / len(latencies)
avg_energy = sum(energies) / len(energies)
```

## Common pitfalls

- Network jitter and server load can significantly skew cloud-based latency measurements.
- Model loading overhead often dominates on-device inference time, masking actual computation speed.
- Framework choice (CPU-only Caffe vs. GPU-enabled CNNDroid) drastically changes on-device performance baselines.

## Evidence (verbatim from paper)

> An empirical evaluation of on-device vs. cloud-based deep inference on mobile devices reveals that on-device CNN inference incurs up to two orders of magnitude higher end-to-end latency and energy consumption due to model loading and computation bottlenecks.

## Citation

```bibtex
@misc{guo2017mobileinference,
  title={Cloud-based or On-device: An Empirical Study of Mobile Deep Inference},
  author={Guo (2017)},
  year={2017},
  note={arXiv:1707.04610}
}
```

- arXiv: 1707.04610