Edge Llm Inference Eval

---
name: edge_llm_inference_eval
description: Evaluates the feasibility and performance of deploying various LLMs on CPU-only edge hardware (Raspberry Pi 5 clusters) by measuring inference speed, resource consumption, and reasoning accuracy under constrained conditions. Use when the user wants to benchmark on OpenAssistant/oasst1 (subset), Winogrande, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.17712
  bibtex_key: nezami2024edgeai
  confidence: high
---

# edge_llm_inference_eval

> Generative AI on the Edge: Architecture and Performance Evaluation — Nezami et al. (2024) (arXiv:2411.17712, 2024)

## What this evaluates

Evaluates the feasibility and performance of deploying various LLMs on CPU-only edge hardware (Raspberry Pi 5 clusters) by measuring inference speed, resource consumption, and reasoning accuracy under constrained conditions.

## Datasets

- **OpenAssistant/oasst1 (subset)** — total 50; splits: test (50); HF `OpenAssistant/oasst1`
- **Winogrande** — total ?; splits: test (-1); HF `automated-research-group/winogrande`

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions on the Winogrande NLI task. The model selects the answer option with the highest log likelihood and compares it to the ground truth label.
- `throughput` — range: tokens/sec
  - Number of tokens generated per second, measured separately for the Prefill and Decode phases.
- `latency` — range: ms or seconds
  - Time per token (ms) and total end-to-end generation time (seconds) from input to output completion.
- `memory_usage` — range: GB
  - Peak RAM consumption during inference, measured in gigabytes.
- `cpu_utilization` — range: percent
  - Average CPU load percentage across all cores during inference.
- `coefficient_of_variation` — range: dimensionless
  - Ratio of the standard deviation to the mean throughput across varying context lengths, used to measure performance stability.

## Input / output format

**Input**: Conversational prompts from OASST1 (average 25 words, range 1–241 words). Maximum token generation is capped at 500.

**Output**: Autoregressive text generation up to 500 tokens. For accuracy evaluation, the model outputs the selected answer option based on highest log likelihood.

## Scoring recipe

```python
def evaluate_throughput_latency(prompt, model, max_tokens=500):
    start = time()
    tokens = model.generate(prompt, max_tokens=max_tokens)
    total_time = time() - start
    throughput = len(tokens) / total_time
    latency_per_token = total_time / len(tokens)
    return throughput, latency_per_token

def evaluate_accuracy(dataset, model):
    correct = 0
    for item in dataset:
        log_probs = [model.log_likelihood(item.context, opt) for opt in item.options]
        predicted = item.options[argmax(log_probs)]
        if predicted == item.label:
            correct += 1
    return correct / len(dataset)

def evaluate_stability(throughputs):
    return std(throughputs) / mean(throughputs)
```

## Common pitfalls

- Quantization significantly impacts accuracy and throughput, but the exact bit-width used for each model is not consistently reported across all experiments.
- Prefill and Decode phases exhibit different performance characteristics; evaluating only end-to-end latency masks phase-specific bottlenecks.
- CPU-only edge hardware (Raspberry Pi 5) lacks GPU acceleration, making results non-transferable to GPU-based edge deployments.

## Evidence (verbatim from paper)

> We assess five key metrics—memory usage, CPU utilization, latency, accuracy, and computational throughput—offering a comprehensive view of resource demands and efficiency for PromptAI in resource-constrained settings.

## Citation

```bibtex
@misc{nezami2024edgeai,
  title={Generative AI on the Edge: Architecture and Performance Evaluation},
  author={Nezami et al. (2024)},
  year={2024},
  note={arXiv:2411.17712}
}
```

- arXiv: 2411.17712