Mlperf Tpu Eval

---
name: mlperf-tpu-eval
description: Evaluates the performance and communication overhead of fault-tolerant 2-D allreduce algorithms compared to standard allreduce during data-parallel ML training on TPU-v3 mesh networks. It measures end-to-end training latency and relative efficiency under simulated chip failure conditions. Use when the user wants to benchmark on MLPerf-v0.7 ResNet-50, MLPerf-v0.7 BERT, or asks about evaluating this task. Reports Relative Efficiency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.03605
  bibtex_key: kumar2020highlyavailable
  confidence: high
---

# mlperf-tpu-eval

> Highly Available Data Parallel ML training on Mesh Networks — Kumar et al. (2020) (arXiv:2011.03605, 2020)

## What this evaluates

Evaluates the performance and communication overhead of fault-tolerant 2-D allreduce algorithms compared to standard allreduce during data-parallel ML training on TPU-v3 mesh networks. It measures end-to-end training latency and relative efficiency under simulated chip failure conditions.

## Datasets

- **MLPerf-v0.7 ResNet-50** — total ?; splits: train (-1)
- **MLPerf-v0.7 BERT** — total ?; splits: train (-1)

## Metrics

- `Relative Efficiency` **(primary)** — range: other
  - Compares normalized throughput between fault-tolerant and full mesh runs while accounting for the reduced number of active chips. Calculated as (Full Mesh Time / Full Mesh Chips) divided by (Fault Tolerant Time / Fault Tolerant Chips).
- `Allreduce Overhead` — range: percent
  - Percentage of the device execution step time consumed by allreduce communication operations.

## Input / output format

**Input**: Distributed model parameters and optimizer states across a 2-D TPU mesh topology, with training batches from ImageNet-1K (ResNet-50) or Wikipedia (BERT).

**Output**: End-to-end benchmark execution time (minutes) and per-step communication overhead percentage measured during device execution.

## Scoring recipe

```python
def compute_relative_efficiency(full_time, full_chips, ft_time, ft_chips):
    throughput_full = full_chips / full_time
    throughput_ft = ft_chips / ft_time
    return throughput_ft / throughput_full

def compute_allreduce_overhead(allreduce_time, total_device_step_time):
    return (allreduce_time / total_device_step_time) * 100

# Evaluation runs simulate a 4x2 failed region (8 chips)
# Metrics are averaged over multiple runs (variance < 2%)
```

## Common pitfalls

- Relative Efficiency normalizes for the reduced chip count in fault-tolerant runs, so a value below 1.0 reflects both algorithmic overhead and hardware reduction, not just communication inefficiency.
- Allreduce Overhead is measured as a fraction of device execution step time, not wall-clock time, which can understate the impact on total training latency.
- Simulated failures are fixed 4x2 rectangular blocks; real-world TPU failure patterns may be irregular and affect routing differently.

## Evidence (verbatim from paper)

> Table 1 shows the end to end time with the two MLPerf benchmarks on 512 and 1024 TPU chips that had 16x32 and 32x32 mesh topologies. The failed region here has a shape of 4x2 with 8 total failed chips. Note, the run-to-run variance here is under 2%. The table also shows the relative efficiency of fault tolerant vs full meshes. The relative efficiency also compensates for the reduction in the number of chips in addition to overheads from the fault tolerant allreduce scheme.

## Citation

```bibtex
@misc{kumar2020highlyavailable,
  title={Highly Available Data Parallel ML training on Mesh Networks},
  author={Kumar et al. (2020)},
  year={2020},
  note={arXiv:2011.03605}
}
```

- arXiv: 2011.03605