Mlperf Hardware Eval

---
name: mlperf-hardware-eval
description: Evaluates the performance and energy efficiency of a proposed deep learning hardware architecture against state-of-the-art GPUs and specialized accelerators. It probes the architecture's ability to handle diverse DL workloads (CNNs, transformers, RNNs) across different batch sizes and its software maturity for general-purpose mapping. Use when the user wants to benchmark on MLPerf benchmark suite, or asks about evaluating this task. Reports Speedup, Rel. Efficiency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.02204
  bibtex_key: davies2021violet
  confidence: high
---

# mlperf-hardware-eval

> Violet: Architecturally Exposed Orchestration, Movement, and Placement for Generalized Deep Learning — Davies et al. (2021) (arXiv:2112.02204, 2021)

## What this evaluates

Evaluates the performance and energy efficiency of a proposed deep learning hardware architecture against state-of-the-art GPUs and specialized accelerators. It probes the architecture's ability to handle diverse DL workloads (CNNs, transformers, RNNs) across different batch sizes and its software maturity for general-purpose mapping.

## Datasets

- **MLPerf benchmark suite** — total ?; splits: test (-1)

## Metrics

- `Speedup` **(primary)** — range: other
  - Ratio of throughput measured in samples/sec between the proposed architecture and the baseline (A100). Higher values indicate better performance.
- `Rel. Efficiency` **(primary)** — range: other
  - Ratio of average energy per operation (pJ/op) between baseline and proposed architecture. Lower pJ/op is better, so efficiency = pJ_op_baseline / pJ_op_proposed.
- `Compute resource utilization` — range: percent
  - Percentage of available compute resources actively used during workload execution.
- `Area efficiency` — range: other
  - Peak compute throughput normalized by die area, measured in TOP/s/mm^2.

## Input / output format

**Input**: MLPerf benchmark suite workloads (Resnet50, SSD-Resnet34, MaskRCNN, Bert Large 128, RNN-T, Alexnet, Mobilenet) executed at small batch size (1) and large batch size (64-128).

**Output**: Throughput (samples/sec), power consumption (W), energy per operation (pJ/op), compute resource utilization (%), and area efficiency (TOP/s/mm^2).

## Scoring recipe

```python
speedup = throughput_proposed / throughput_baseline
rel_efficiency = pJ_op_baseline / pJ_op_proposed
geo_mean_speedup = exp(mean(log(speedup)))
geo_mean_efficiency = exp(mean(log(rel_efficiency)))
compute_utilization = (active_cycles / total_cycles) * 100
area_efficiency = peak_tops / die_area_mm2
```

## Common pitfalls

- Hardware simulation results may differ from real silicon due to unmodeled memory latency and mapping overheads.
- Performance is highly sensitive to batch size; small batches favor the proposed architecture while large batches narrow the gap.
- Compute utilization metrics are heavily dependent on architecture-specific compiler mapping strategies and may not reflect raw hardware capability.

## Evidence (verbatim from paper)

> Speedup is measured by the ratio of throughput in samples/sec, and Rel. Efficiency is measured by the ratio of average pJ/op for the network. At large batch, UPCYCLE provides modest performance improvements over A100 - geo-mean 2.2X and 2.3X for inference and training performance respectively - as well as 6X power efficiency improvement in both cases.

## Citation

```bibtex
@misc{davies2021violet,
  title={Violet: Architecturally Exposed Orchestration, Movement, and Placement for Generalized Deep Learning},
  author={Davies et al. (2021)},
  year={2021},
  note={arXiv:2112.02204}
}
```

- arXiv: 2112.02204