Cv Inference Eval

---
name: cv-inference-eval
description: Evaluates end-to-end inference latency and hardware efficiency of computer vision models on edge AI hardware. It probes how well a hardware-software co-design optimizes data movement and compute utilization under strict memory and bandwidth constraints. Use when the user wants to benchmark on ImageNet, COCO 2017, or asks about evaluating this task. Reports Latency [ms].
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.14388
  bibtex_key: bamberg2025eiqneutron
  confidence: high
---

# cv-inference-eval

> eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations — Bamberg et al. (2025) (arXiv:2509.14388, 2025)

## What this evaluates

Evaluates end-to-end inference latency and hardware efficiency of computer vision models on edge AI hardware. It probes how well a hardware-software co-design optimizes data movement and compute utilization under strict memory and bandwidth constraints.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **COCO 2017** — total ?; splits: test (-1)

## Metrics

- `Latency [ms]` **(primary)** — range: other
  - Batch-size-1 end-to-end inference time measured on silicon. Lower values indicate faster execution.
- `LTP` — range: other
  - Latency-TOPS Product, defined as latency multiplied by hardware TOPS. Lower values indicate higher efficiency, meaning less silicon is required to achieve a given performance.

## Input / output format

**Input**: INT8-quantized computer vision model (classification, detection, or segmentation) exported to LiteRT v2.18.0, executed on a production SoC with 2 TOPS NPU, 12 GB/s DDR bandwidth, and 1 MB SRAM.

**Output**: End-to-end inference latency in milliseconds and the computed Latency-TOPS Product (LTP) metric.

## Scoring recipe

```python
def evaluate(model, hardware_top_s):
    # Measure batch-size-1 end-to-end inference time on silicon
    latency_ms = measure_batch1_inference_time(model, hardware_config)
    # Compute efficiency metric
    ltp = latency_ms * hardware_top_s
    return {"latency_ms": latency_ms, "ltp": ltp}
```

## Common pitfalls

- iNPU latency is approximated as the inverse of reported throughput, which is a theoretical lower bound that may overestimate actual performance due to pipeline depth.
- Models are strictly INT8 quantized; accuracy is not reported, so comparisons rely solely on latency and efficiency metrics.
- Comparing systems with different TOPS and bandwidth requires using LTP rather than raw latency to normalize for hardware capability.

## Evidence (verbatim from paper)

> Real-life (silicon) performance is measured as batch-size-1 end-to-end inference latency on a production, mass-market micro-processor-unit SoC (MPU (micro-processor-unit SoC))–embedding the proposed 2-TOPS NPU with 1MiB of SRAM and 12GB/s of DDR bandwidth—employing LiteRT benchmarking tools. To capture performance efficiency—and thereby power and area—we introduce a second metric in addition to latency: the Latency–TOPS Product (LTP), defined as: LTP ≜ latency·TOPS, where lower values indicate higher efficiency, meaning less silicon hardware is required to achieve a given performance.

## Citation

```bibtex
@misc{bamberg2025eiqneutron,
  title={eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations},
  author={Bamberg et al. (2025)},
  year={2025},
  note={arXiv:2509.14388}
}
```

- arXiv: 2509.14388