Migperf Eval

---
name: migperf-eval
description: Evaluates deep learning training and inference workloads on NVIDIA's Multi-Instance GPU (MIG) technology. It systematically measures performance trade-offs across different partition sizes, batch sizes, and model architectures, while comparing MIG against software-based GPU sharing (MPS) and testing framework compatibility. Use when the user wants to benchmark on Representative DL Models (ResNet18, ResNet50, BERT, ViT, Diffusion), or asks about evaluating this task. Reports tail latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.00407
  bibtex_key: zhang2023migperf
  confidence: high
---

# migperf-eval

> MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs — Huaizheng Zhang et al. (2023) (arXiv:2301.00407, 2023)

## What this evaluates

Evaluates deep learning training and inference workloads on NVIDIA's Multi-Instance GPU (MIG) technology. It systematically measures performance trade-offs across different partition sizes, batch sizes, and model architectures, while comparing MIG against software-based GPU sharing (MPS) and testing framework compatibility.

## Datasets

- **Representative DL Models (ResNet18, ResNet50, BERT, ViT, Diffusion)** — total ?; splits: test (-1); repo https://github.com/MLSysOps/MIGProfiler

## Metrics

- `average latency` — range: ms
  - Average processing time per batch of requests for inference workloads.
- `tail latency` **(primary)** — range: ms
  - X-th percentile (e.g., 99th) of request latencies, measuring worst-case processing time.
- `throughput` — range: req/min
  - Number of training samples or inference requests processed per unit time (e.g., per minute).
- `GRACT` — range: [0, 1]
  - Ratio of allocated computation resources to total available resources on a GPU instance.
- `Frame Buffer` — range: GB
  - Occupied GPU memory during workload execution.
- `Energy Consumption` — range: other
  - Estimated electricity used for running a workload over a specific period (e.g., 5 minutes).

## Input / output format

**Input**: Deep learning training or inference workloads specified by model architecture, batch size, and MIG partition configuration (e.g., 1g.10gb).

**Output**: Performance metrics (latency, throughput, utilization, memory, energy) and framework compatibility status (detected/not detected).

## Scoring recipe

```python
for each workload_config in configs:
    start_timer()
    run_inference_or_training(workload_config)
    end_timer()
    metrics[workload_config] = {
        'avg_latency': mean(request_latencies),
        'tail_latency': percentile(request_latencies, 99),
        'throughput': count_requests / duration_minutes,
        'gract': compute_gpu_utilization(),
        'fb': measure_gpu_memory(),
        'energy': estimate_energy_consumption(duration_minutes)
    }
return metrics
```

## Common pitfalls

- Performance comparison between MIG and MPS is highly sensitive to batch size; small batches show negligible differences, while large batches reveal MIG's tail latency advantage.
- Framework compatibility tests only check if frameworks can detect multiple GIs; they do not evaluate actual performance on non-zero GIs due to driver limitations.
- Energy consumption is driven by processing speed rather than just batch size; larger instances may consume less energy for the same workload due to faster completion.

## Evidence (verbatim from paper)

> Latency measures the processing speed. We use two metrics, average latency and tail latency for inference workloads. Average latency measures the average processing time for a batch of requests, while tail latency means that the X-th (e.g. 99) percentile of requests has lower latency than the given value. Throughput measures the training samples or inference requests that a device can process within a time unit (e.g., 1 min).

## Citation

```bibtex
@misc{zhang2023migperf,
  title={MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs},
  author={Huaizheng Zhang et al. (2023)},
  year={2023},
  note={arXiv:2301.00407}
}
```

- arXiv: 2301.00407