Edge Ai Platform Inference Eval

---
name: edge-ai-platform-inference-eval
description: Evaluates the inference performance of heterogeneous edge AI platforms (CPU, GPU, NPU) across fundamental linear algebra primitives and diverse neural network models. It probes hardware efficiency in compute-bound versus memory-bound workloads, batch processing scalability, and quantization support. Use when the user wants to benchmark on Matrix Multiplication, Matrix-Vector Multiplication, Dot Product, MobileNetV2, LSTM, TinyLlama, or asks about evaluating this task. Reports Latency (ms).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.14803
  bibtex_key: jayanth2024edgeai
  confidence: high
---

# edge-ai-platform-inference-eval

> Benchmarking Edge AI Platforms for High-Performance ML Inference — Jayanth et al. (2024) (arXiv:2409.14803, 2024)

## What this evaluates

Evaluates the inference performance of heterogeneous edge AI platforms (CPU, GPU, NPU) across fundamental linear algebra primitives and diverse neural network models. It probes hardware efficiency in compute-bound versus memory-bound workloads, batch processing scalability, and quantization support.

## Datasets

- **Matrix Multiplication** — total ?; splits: (unstated)
- **Matrix-Vector Multiplication** — total ?; splits: (unstated)
- **Dot Product** — total ?; splits: (unstated)
- **MobileNetV2** — total ?; splits: (unstated)
- **LSTM** — total ?; splits: (unstated)
- **TinyLlama** — total ?; splits: (unstated)

## Metrics

- `Latency (ms)` **(primary)** — range: other
  - Total time taken to generate output from when an input is provided, measured in milliseconds.
- `Avg Throughput (GFLOPS)` — range: other
  - Average number of floating-point operations performed per second, calculated as total FLOPs divided by execution time, measured in GigaFLOPS.
- `FPS` — range: other
  - Frames Per Second; the number of individual images processed per second, used specifically for video classification throughput.

## Input / output format

**Input**: Linear algebra: square matrices A (N×N) and B (N×N), or vectors x (N×1) and y (N×1) with N ranging from 16 to 1024. Neural networks: video frames (MobileNetV2), time-series sequences (LSTM), and text tokens (TinyLlama). Batch sizes are varied for video classification tasks.

**Output**: Standard model predictions/classifications and hardware performance measurements (latency in ms, throughput in GFLOPS, or FPS). No custom output formatting is required beyond standard inference results.

## Scoring recipe

```python
def compute_metrics(execution_times_ms, total_flops, batch_size, task_type):
    avg_latency_ms = sum(execution_times_ms) / len(execution_times_ms)
    avg_throughput_gflops = (total_flops / (avg_latency_ms / 1000.0)) / 1e9
    fps = None
    if task_type == 'video_classification':
        fps = batch_size / (avg_latency_ms / 1000.0)
    return {'latency_ms': avg_latency_ms, 'throughput_gflops': avg_throughput_gflops, 'fps': fps}
```

## Common pitfalls

- Precision mismatch: The paper exclusively uses FP-16 for uniform analysis across platforms, despite CPUs/GPUs natively supporting FP-32.
- Framework overhead: OpenVINO and IPEX use lazy evaluation and asynchronous execution, which can skew latency measurements if warm-up runs are omitted or if results are not properly synchronized.
- Batch size sensitivity: Performance characteristics shift dramatically between compute-bound (high batch) and memory-bound (low batch) regimes, requiring explicit reporting of batch sizes for reproducibility.

## Evidence (verbatim from paper)

> The metrics we use for these operations are latency and average throughput. For LSTM and LLM models, inference latency will be the performance metric, and for video classification, along with inference latency, FPS (Frames Per Second) will be used as a measure of throughput.

## Citation

```bibtex
@misc{jayanth2024edgeai,
  title={Benchmarking Edge AI Platforms for High-Performance ML Inference},
  author={Jayanth et al. (2024)},
  year={2024},
  note={arXiv:2409.14803}
}
```

- arXiv: 2409.14803