Llama Energy Latency Eval

---
name: llama-energy-latency-eval
description: This benchmark evaluates the inference latency and energy consumption of LLaMA models (7B-65B) across different GPU hardware (V100, A100) and sharding configurations. It probes the trade-offs between computational throughput, power usage, and hardware efficiency during text generation. Use when the user wants to benchmark on Alpaca, GSM8K, or asks about evaluating this task. Reports energy per second (Watts).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.03003
  bibtex_key: samsi2023words
  confidence: high
---

# llama-energy-latency-eval

> From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Samsi et al. (2023) (arXiv:2310.03003, 2023)

## What this evaluates

This benchmark evaluates the inference latency and energy consumption of LLaMA models (7B-65B) across different GPU hardware (V100, A100) and sharding configurations. It probes the trade-offs between computational throughput, power usage, and hardware efficiency during text generation.

## Datasets

- **Alpaca** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)

## Metrics

- `words per second` — range: other
  - Total number of words generated divided by the total inference time in seconds.
- `tokens per second` — range: other
  - Total number of tokens generated divided by the total inference time in seconds.
- `responses per second` — range: other
  - Total number of completed responses divided by the total inference time in seconds.
- `energy per second (Watts)` **(primary)** — range: other
  - Average power draw measured in Watts during inference, calculated as total energy consumed divided by inference duration.
- `energy per output token (Joules)` — range: other
  - Total energy consumed in Joules divided by the total number of output tokens generated.
- `energy per response (Joules)` — range: other
  - Total energy consumed in Joules divided by the total number of responses generated.

## Input / output format

**Input**: Text prompts from the Alpaca and GSM8K datasets.

**Output**: Generated text sequences up to a specified maximum generation length (e.g., 256, 512, 1024 tokens).

## Scoring recipe

```python
def calculate_metrics(total_words, total_tokens, total_responses, total_time_sec, total_energy_joules):
    wps = total_words / total_time_sec
    tps = total_tokens / total_time_sec
    rps = total_responses / total_time_sec
    watts = total_energy_joules / total_time_sec
    j_per_token = total_energy_joules / total_tokens
    j_per_resp = total_energy_joules / total_responses
    return {'wps': wps, 'tps': tps, 'rps': rps, 'watts': watts, 'j_per_token': j_per_token, 'j_per_resp': j_per_resp}
```

## Common pitfalls

- Energy per second (Watts) increases with more GPUs/shards due to baseline power draw, even if throughput improves.
- Power capping reduces energy but increases latency non-linearly; optimal cap depends on workload and hardware.
- Memory utilization is often low (23-27%) for sharded models, leaving room for co-location but not directly reducing per-model energy.

## Evidence (verbatim from paper)

> Figure 2 shows a baseline comparison of inference performance of the three LLaMA variants on both the V100 and A100 GPUs respectively. For each model, in line with the spirit of the bare minimum settings, inference is done with a batch size of 64 and an maximum generation length of 256. ... As expected, we observe that the A100 outperforms V100 on both the Alpaca and GSM8K datasets: particularly for the smaller LLaMA 7B and 13B, we see anywhere from a 2 times (7B) to a 1.25 times increase (13B) in inference latency on the A100 when compared to the V100 across words per second, tokens per second, and responses per second. Figure 3 shows a comparison of the energy per second required to run inference on LLaMA 7B, 13B, and 65B, with different GPUs under the same bare minimum hardware settings as the above.

## Citation

```bibtex
@misc{samsi2023words,
  title={From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference},
  author={Samsi et al. (2023)},
  year={2023},
  note={arXiv:2310.03003}
}
```

- arXiv: 2310.03003