Mnn Inference Benchmark Eval

---
name: mnn-inference-benchmark-eval
description: Evaluates the inference latency and computational efficiency of mobile deep learning engines across different hardware platforms, compute backends, and neural network architectures. Use when the user wants to benchmark on MobileNet-v1, SqueezeNet-v1.1, ResNet-18, Inception-v3, or asks about evaluating this task. Reports inference time (ms).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2002.12418
  bibtex_key: jiang2020mnn
  confidence: high
---

# mnn-inference-benchmark-eval

> MNN: A Universal and Efficient Inference Engine — Jiang et al. (2020) (arXiv:2002.12418, 2020)

## What this evaluates

Evaluates the inference latency and computational efficiency of mobile deep learning engines across different hardware platforms, compute backends, and neural network architectures.

## Datasets

- **MobileNet-v1** — total ?; splits: test (-1)
- **SqueezeNet-v1.1** — total ?; splits: test (-1)
- **ResNet-18** — total ?; splits: test (-1)
- **Inception-v3** — total ?; splits: test (-1)

## Metrics

- `inference time (ms)` **(primary)** — range: other
  - Average latency in milliseconds to process a single 224×224 RGB image (batch size 1) over 10 runs, preceded by one warm-up inference.

## Input / output format

**Input**: Single 224×224 RGB image

**Output**: Model predictions (evaluation measures execution latency rather than output format or correctness)

## Scoring recipe

```python
warmup_run(model, input_image)
latencies = []
for _ in range(10):
    start = time()
    model(input_image)
    latencies.append(time() - start)
return sum(latencies) / len(latencies)
```

## Common pitfalls

- Omitting the mandatory warm-up inference run before benchmarking
- Failing to average over exactly 10 runs or using a different batch size/input resolution
- Comparing GPU backends without specifying the underlying API (Metal, OpenCL, OpenGL, or Vulkan)

## Evidence (verbatim from paper)

> We report the inference time of one 224 × 224 RGB image (i.e., batch size is 1), averaged by 10 runs. Before benchmark, one warm-up inference is conducted for fair comparison with other works (Tencent, 2017; Google, 2017a).

## Citation

```bibtex
@misc{jiang2020mnn,
  title={MNN: A Universal and Efficient Inference Engine},
  author={Jiang et al. (2020)},
  year={2020},
  note={arXiv:2002.12418}
}
```

- arXiv: 2002.12418