Mlperf Inference Eval

---
name: mlperf-inference-eval
description: Evaluates ML inference systems across diverse hardware and software stacks under realistic deployment scenarios. It measures both model quality against strict baselines and system performance (latency/throughput) to enable architecture-neutral comparisons of production-like workloads. Use when the user wants to benchmark on ImageNet, COCO, WMT16 EN-DE, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.02549
  bibtex_key: reddi2019mlperf
  confidence: high
---

# mlperf-inference-eval

> MLPerf Inference Benchmark — Vijay Janapa Reddi et al. (arXiv:1911.02549, 2019)

## What this evaluates

Evaluates ML inference systems across diverse hardware and software stacks under realistic deployment scenarios. It measures both model quality against strict baselines and system performance (latency/throughput) to enable architecture-neutral comparisons of production-like workloads.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **COCO** — total ?; splits: test (-1)
- **WMT16 EN-DE** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels out of total samples. MLPerf requires achieving 99% of the FP32 baseline accuracy before system metrics are recorded.
- `mAP` — range: [0, 1]
  - Mean Average Precision for object detection bounding boxes. Must reach 99% of the FP32 baseline (e.g., 0.20) to qualify for latency measurement.
- `SacreBleu` — range: [0, 100]
  - BLEU score computed with standardized tokenization and reference handling. Must reach 99% of the FP32 baseline (e.g., 23.9) to qualify.

## Input / output format

**Input**: Image tensors (224x224, 300x300, or 1200x1200) or text sequences for machine translation.

**Output**: Predicted class labels, bounding boxes with confidence scores, or translated text sequences.

## Scoring recipe

```python
def evaluate(predictions, gold, baseline_accuracy):
    if metric == 'Top-1 accuracy':
        score = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif metric == 'mAP':
        score = compute_map(predictions, gold)
    elif metric == 'SacreBleu':
        score = sacrebleu.corpus_bleu(predictions, [gold])
    
    # Quality gate: must meet 99% of FP32 baseline
    if score < 0.99 * baseline_accuracy:
        return {'valid': False, 'score': score}
    
    # Measure latency/throughput per deployment scenario
    latency = measure_latency(predictions, gold)
    return {'valid': True, 'accuracy': score, 'latency': latency}
```

## Common pitfalls

- Failing the quality gate: latency/throughput measurements are invalid if the model does not first achieve 99% of the FP32 baseline accuracy/mAP/SacreBleu.
- Ignoring scenario constraints: single-stream, multistream, server, and offline scenarios enforce strict data-availability and batching rules that drastically change system performance.
- Framework-specific operator differences (e.g., NMS variants) can cause accuracy drops during model conversion, requiring careful validation before benchmarking.

## Evidence (verbatim from paper)

> These models vary tremendously in compute and memory requirements (e.g., a 50× difference in gigaflops), while the corresponding Top-1 accuracy ranges from 55% to 83%.

## Citation

```bibtex
@misc{reddi2019mlperf,
  title={MLPerf Inference Benchmark},
  author={Vijay Janapa Reddi et al.},
  year={2019},
  note={arXiv:1911.02549}
}
```

- arXiv: 1911.02549