Edge Llm Inference Benchmark

---
name: edge-llm-inference-benchmark
description: Evaluates the trade-offs between token throughput, latency, energy efficiency, and physical footprint when deploying compact LLMs on various IoT-grade single-board computers with different hardware accelerators (CPU, NPU, GPU). Use when the user has predictions and gold and needs to compute Throughput (tokens/s), Time-to-first-token (TTFT), Energy per million tokens (MJ/Mtok).
metadata:
  skill_kind: metric
  source_arxiv: 2604.24785
  bibtex_key: [REDACTED]
  confidence: high
---

# edge-llm-inference-benchmark

> Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers — Renney et al. (2026) (arXiv:2604.24785, 2026)

## What this evaluates

Evaluates the trade-offs between token throughput, latency, energy efficiency, and physical footprint when deploying compact LLMs on various IoT-grade single-board computers with different hardware accelerators (CPU, NPU, GPU).

## Datasets

- **Static Prompt Benchmark** — total 1; splits: test (1); repo https://github.com/SquidyBallinx11011/LLM-Edge-Benchmarking-Suite

## Metrics

- `Throughput (tokens/s)` **(primary)** — range: tokens/s
  - Total tokens generated divided by the total elapsed wall-clock time for the generation run.
- `Time-to-first-token (TTFT)` **(primary)** — range: seconds
  - Wall-clock time elapsed between prompt submission and the receipt of the first generated token.
- `Energy per million tokens (MJ/Mtok)` **(primary)** — range: MJ/Mtok
  - Total energy consumed (derived from power meter readings and generation time) scaled to represent the energy required to generate one million tokens.
- `Throughput density (Tps/m3)` — range: Tps/m3
  - Token throughput per second normalised by the physical device volume in cubic metres, quantifying inference capability per unit of space.

## Input / output format

**Input**: Fixed prompt: "Explain why the sky is blue in two or more paragraphs." with generation length capped at 100 tokens via num_predict.

**Output**: Streaming text generation, up to 100 tokens, returned incrementally as they are generated.

## Scoring recipe

```python
def compute_metrics(wall_start, first_token_time, wall_end, tokens_count, power_watts, device_volume_m3):
    total_time = wall_end - wall_start
    throughput = tokens_count / total_time
    ttft = first_token_time - wall_start
    energy_joules = power_watts * total_time
    energy_per_mtok = (energy_joules / 1_000_000) / (tokens_count / 1_000_000)
    throughput_density = throughput / device_volume_m3
    return throughput, ttft, energy_per_mtok, throughput_density

# Average results over n=5 runs after a warmup request
```

## Common pitfalls

- Failing to issue a warmup request before timed runs, which skews initial throughput and TTFT due to model loading overhead.
- Comparing raw energy consumption without scaling to a fixed token count (e.g., 1M tokens), which unfairly penalizes slower hardware or shorter generation runs.
- Ignoring quantization precision differences across runtimes (e.g., Q4_K_M on CPU vs INT8/INT4 on NPUs/GPUs), which significantly alters accelerator utilization and power draw.

## Evidence (verbatim from paper)

> Performance was evaluated using three primary metrics: Throughput (tokens/s): the rate of token generation during inference. Time-to-first-token (TTFT): latency between prompt submission and generation of the first token. Energy consumption (MJ/Mtok): energy required to generate one million tokens, capturing hardware efficiency.

## Citation

```bibtex
@misc{renney2026cloudtoedge,
  title={Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers},
  author={Renney et al. (2026)},
  year={2026},
  note={arXiv:2604.24785}
}
```

- arXiv: 2604.24785