Milabench Eval

---
name: milabench-eval
description: Evaluates the real-world computational performance and software stack maturity of AI accelerators across diverse workloads including NLP, computer vision, reinforcement learning, and graph neural networks. Use when the user wants to benchmark on Milabench, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.11940
  bibtex_key: [REDACTED]
  confidence: medium
---

# milabench-eval

> Introducing Milabench: Benchmarking Accelerators for AI — Delaunay et al. (2024) (arXiv:2411.11940, 2024)

## What this evaluates

Evaluates the real-world computational performance and software stack maturity of AI accelerators across diverse workloads including NLP, computer vision, reinforcement learning, and graph neural networks.

## Datasets

- **Milabench** — total ?; splits: (unstated)

## Metrics

- `performance` **(primary)** — range: other
  - Throughput or execution speed of a workload on a target GPU divided by the same workload's performance on the NVIDIA A100 baseline. Values >1 indicate faster execution or higher throughput than the baseline.

## Input / output format

**Input**: Standardized end-to-end AI workloads (training, fine-tuning, or inference) implemented in PyTorch/JAX, including model definitions, data loaders, and hyperparameters, executed on the target GPU node.

**Output**: Execution throughput (e.g., tokens/sec, images/sec, or FLOPS) and relative performance compared to the A100 baseline.

## Scoring recipe

```python
def compute_performance(target_perf, baseline_perf):
    if baseline_perf == 0:
        return float('inf')
    return target_perf / baseline_perf
# Aggregate across workloads by averaging relative_performance values
```

## Common pitfalls

- Synthetic FLOP benchmarks (e.g., peak TF32/FP16) often fail to predict real-world workload performance due to software stack maturity and kernel optimization differences.
- Vendor default precision settings (e.g., Gaudi2 defaulting to TF32) can mask true hardware capabilities in standard benchmarks.
- Framework and library updates (e.g., new PyTorch versions) frequently introduce performance gains that vary by vendor, making results time-sensitive.

## Evidence (verbatim from paper)

> The performance results from the A100 were used as the baseline for comparison with the H100, MI300X, and Gaudi2 GPUs.

## Citation

```bibtex
@misc{delaunay2024milabench,
  title={Introducing Milabench: Benchmarking Accelerators for AI},
  author={Delaunay et al. (2024)},
  year={2024},
  note={arXiv:2411.11940}
}
```

- arXiv: 2411.11940