Llm Npu Eval

---
name: llm-npu-eval
description: This evaluation protocol assesses the performance of an on-device LLM inference framework (llm.npu) on mobile NPUs. It probes the system's ability to accelerate prefill and decoding stages, manage quantization without accuracy loss, and optimize energy efficiency across various LLM sizes and real-world task datasets. Use when the user wants to benchmark on LAMBADA, HellaSwag, WinoGrande, MMLU, LongBench, DroidTask, Persona-Chat, or asks about evaluating this task. Reports end-to-end latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.05858
  bibtex_key: xu2024fastondevice
  confidence: high
---

# llm-npu-eval

> Fast On-device LLM Inference with NPUs — Xu et al. (2024) (arXiv:2407.05858, 2024)

## What this evaluates

This evaluation protocol assesses the performance of an on-device LLM inference framework (llm.npu) on mobile NPUs. It probes the system's ability to accelerate prefill and decoding stages, manage quantization without accuracy loss, and optimize energy efficiency across various LLM sizes and real-world task datasets.

## Datasets

- **LAMBADA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **LongBench** — total ?; splits: test (-1)
- **DroidTask** — total ?; splits: test (-1)
- **Persona-Chat** — total ?; splits: test (-1)

## Metrics

- `end-to-end latency` **(primary)** — range: seconds
  - Total time from prompt input to final token generation completion, measured in seconds.
- `prefill latency` — range: seconds
  - Time taken to process the input prompt and compute KV cache, measured in seconds.
- `prefill energy consumption` — range: joules
  - Energy used during the prefill stage, measured via Android OS power supply interface profiling every 100ms.
- `accuracy` — range: percent
  - Percentage of correctly answered questions across multiple-choice and open-ended benchmarks.

## Input / output format

**Input**: Text prompts of varying lengths (64, 256, 1024 tokens for controlled tests; dataset-specific lengths for real-world tasks) fed to the LLM.

**Output**: Generated text sequences with specified output lengths (e.g., 2–4, 5–11, 1–5, 3–5, or 35–57 tokens depending on the dataset).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_latency(start_time, end_time):
    return end_time - start_time  # seconds

def compute_energy(power_samples, interval=0.1):
    return sum(power_samples) * interval  # joules
```

## Common pitfalls

- llm.npu uses a CPU backend for decoding, which can bottleneck end-to-end latency compared to baselines that optimize GPU decoding.
- Energy consumption measurements are only available on rootable devices (Redmi K60 Pro), limiting cross-device energy comparisons.
- Shorter prompts (e.g., 64 tokens) cause padding issues and reduce NPU scheduling efficiency, leading to lower speedups than longer prompts.

## Evidence (verbatim from paper)

> We mainly measure LLM inference accuracy, prefill latency, prefill energy consumption, prefill memory consumption and end-to-end inference latency. The energy consumption is obtained through /sys/class/power_supply in Android OS by profiling every 100ms. All experiments are repeated three times and we report the average numbers.

## Citation

```bibtex
@misc{xu2024fastondevice,
  title={Fast On-device LLM Inference with NPUs},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2407.05858}
}
```

- arXiv: 2407.05858