Mlperf Power Eval

---
name: mlperf-power-eval
description: Evaluates the energy efficiency of machine learning systems across diverse hardware scales (data center, edge, tiny) and workloads (inference and training). It measures how effectively systems convert electrical energy into computational progress, tracking improvements in samples processed per joule over time and across system configurations. Use when the user wants to benchmark on MLPerf, or asks about evaluating this task. Reports samples per joule.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12032
  bibtex_key: [REDACTED]
  confidence: high
---

# mlperf-power-eval

> MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Tschand et al. (2024) (arXiv:2410.12032, 2024)

## What this evaluates

Evaluates the energy efficiency of machine learning systems across diverse hardware scales (data center, edge, tiny) and workloads (inference and training). It measures how effectively systems convert electrical energy into computational progress, tracking improvements in samples processed per joule over time and across system configurations.

## Datasets

- **MLPerf** — total 1841; splits: datacenter (590), edge (792), tiny (447), training (12); repo https://github.com/mlcommons/policies

## Metrics

- `samples per joule` **(primary)** — range: other
  - Total number of inference samples (or tokens converted to samples) processed divided by the total energy consumed in joules. Often reported inversely as Joules per Sample (J/Sample) for direct energy cost comparison.
- `time to solution` — range: other
  - Wall-clock time required to complete the training or inference workload.

## Input / output format

**Input**: System configuration (hardware, accelerators, interconnects), workload definition (model architecture, dataset, batch size, precision), and power/energy logs from verified MLPerf submission runs.

**Output**: Normalized energy efficiency (samples per joule), absolute energy consumption (J/Sample), and time-to-solution metrics.

## Scoring recipe

```python
def compute_energy_efficiency(total_samples, total_energy_joules):
    if total_energy_joules <= 0:
        return 0.0
    return total_samples / total_energy_joules
# For LLMs, convert tokens to samples using median sequence length
# samples = tokens / median_sequence_length
```

## Common pitfalls

- Comparing energy efficiency across different hardware scales (datacenter vs edge vs tiny) without accounting for workload complexity and input size differences.
- Using inconsistent performance metrics for LLMs (samples/second vs tokens/second) when calculating energy efficiency, leading to non-comparable J/Sample values.
- Ignoring interconnect and networking energy costs when scaling training systems, which significantly impacts total energy consumption despite reduced wall-clock time.

## Evidence (verbatim from paper)

> These advancements are expressed using normalized samples per joule, a measure that encapsulates enhancements in processing power alongside an industry-wide emphasis on optimizing AI technology for energy efficiency.

## Citation

```bibtex
@misc{tschand2024mlperfpower,
  title={MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI},
  author={Tschand et al. (2024)},
  year={2024},
  note={arXiv:2410.12032}
}
```

- arXiv: 2410.12032