Mlperf Bert Eval

---
name: mlperf-bert-eval
description: Evaluates the distributed training efficiency of the BERT-Large model on variable-length masked language modeling tasks. It measures how quickly the model converges to a target accuracy and the sustained throughput achieved across multiple GPUs. Use when the user wants to benchmark on MLPerf BERT, or asks about evaluating this task. Reports Time to 72% MLM accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2208.08124
  bibtex_key: zeng2022boosting
  confidence: high
---

# mlperf-bert-eval

> Boosting Distributed Training Performance of the Unpadded BERT Model — Zeng et al. (2022) (arXiv:2208.08124, 2022)

## What this evaluates

Evaluates the distributed training efficiency of the BERT-Large model on variable-length masked language modeling tasks. It measures how quickly the model converges to a target accuracy and the sustained throughput achieved across multiple GPUs.

## Datasets

- **MLPerf BERT** — total ?; splits: train (-1), eval (-1)

## Metrics

- `Time to 72% MLM accuracy` **(primary)** — range: minutes
  - Wall-clock time in minutes required for the model to reach 72% Masked Language Modeling accuracy on the held-out evaluation dataset during pre-training.
- `Throughput (Samples/s)` — range: samples/s
  - Number of training samples processed per second, calculated as total batch size divided by training time.

## Input / output format

**Input**: Variable-length tokenized sequences for Masked Language Modeling (MLM) pre-training, processed without padding overhead.

**Output**: Aggregated performance metrics: convergence time in minutes and training throughput in samples per second.

## Scoring recipe

```python
def compute_throughput(total_samples, total_time_seconds):
    return total_samples / total_time_seconds

def compute_convergence_time(eval_dataset, model, target_acc=0.72):
    start_time = current_time()
    for batch in eval_dataset:
        acc = evaluate_mlm_accuracy(model, batch)
        if acc >= target_acc:
            return current_time() - start_time
    return float('inf')
```

## Common pitfalls

- Hardware is strictly fixed to 8x NVIDIA A100 400W GPUs; scaling to different GPU counts or memory configurations will yield different speedups.
- Global batch size is fixed at 448 (56 per GPU); changing it alters both throughput and convergence dynamics.
- Unpadded computation eliminates padding token overhead, making direct comparisons with padded baselines (e.g., DeepSpeed) misleading without normalizing for effective sequence length.

## Evidence (verbatim from paper)

> As shown in Table [III], our work can achieve the fastest throughput with roughly 2578 Samples/s. The metrics for the MLPerf BERT model is the time to train that converges to 72% MLM (Masked Language Modeling) accuracy on the evaluation data set.

## Citation

```bibtex
@misc{zeng2022boosting,
  title={Boosting Distributed Training Performance of the Unpadded BERT Model},
  author={Zeng et al. (2022)},
  year={2022},
  note={arXiv:2208.08124}
}
```

- arXiv: 2208.08124