Ml Accelerator Inference Eval

---
name: ml-accelerator-inference-eval
description: Evaluates the inference latency, power consumption, and computational throughput of various machine learning accelerators and CPUs on object detection tasks. It probes the real-world SWaP (Size, Weight, and Power) efficiency and performance discrepancies between advertised and actual hardware capabilities. Use when the user wants to benchmark on Microsoft COCO, or asks about evaluating this task. Reports Avg. Single Image Inference Time (ms).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.11348
  bibtex_key: [REDACTED]
  confidence: high
---

# ml-accelerator-inference-eval

> Survey and Benchmarking of Machine Learning Accelerators — Reuther et al. (2019) (arXiv:1908.11348, 2019)

## What this evaluates

Evaluates the inference latency, power consumption, and computational throughput of various machine learning accelerators and CPUs on object detection tasks. It probes the real-world SWaP (Size, Weight, and Power) efficiency and performance discrepancies between advertised and actual hardware capabilities.

## Datasets

- **Microsoft COCO** — total ?; splits: test (-1)

## Metrics

- `Avg. Single Image Inference Time (ms)` **(primary)** — range: other
  - Average time in milliseconds to process a single image through the neural network.
- `Measured GOPS/W` — range: other
  - Measured computational throughput in giga operations per second divided by measured power consumption in watts.
- `Avg. Model Load Time (s)` — range: other
  - Average time in seconds to load the entire neural network model onto the device for each image.
- `Measured Power (W)` — range: other
  - Average power draw in watts measured via USB multimeter during inference.

## Input / output format

**Input**: Single images from the Microsoft COCO dataset, processed sequentially with batch size = 1.

**Output**: Inference predictions (object detection outputs), though the benchmark primarily records latency and power metrics rather than prediction accuracy.

## Scoring recipe

```python
for each device:
  load_model(device)
  total_time = 0
  total_power = 0
  for image in dataset:
    start = time.now()
    predict = run_inference(device, image)
    end = time.now()
    total_time += (end - start)
    total_power += measure_power_via_multimeter(device)
  avg_inference_ms = (total_time / N) * 1000
  avg_power_w = total_power / N
  gops_w = measured_gops / avg_power_w
```

## Common pitfalls

- Software throttles enforce batch size = 1 and require full model reload per image, which underestimates production performance.
- Power is measured via USB multimeter rather than system-level power draw, potentially missing overhead from host CPU/memory.
- Reported hardware specs (GOPS, GOPS/W) often significantly overstate actual measured performance, especially for NCS2.

## Evidence (verbatim from paper)

> Table I summarizes the reported and measured giga operations per second (GOPS), power (W), and GOPS/W along with average model load time in seconds and average single image inference time in milliseconds.

## Citation

```bibtex
@misc{reuther2019survey,
  title={Survey and Benchmarking of Machine Learning Accelerators},
  author={Reuther et al. (2019)},
  year={2019},
  note={arXiv:1908.11348}
}
```

- arXiv: 1908.11348