Mlperf Hpc Eval

---
name: mlperf-hpc-eval
description: Evaluates end-to-end performance of HPC systems for scientific machine learning, focusing on data staging, I/O efficiency, and model convergence under massive dataset constraints. It measures how well systems handle large-scale volumetric and high-resolution image workloads while meeting strict accuracy targets. Use when the user wants to benchmark on CosmoFlow, DeepCAM, or asks about evaluating this task. Reports MAE, IOU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.11466
  bibtex_key: farrell2021mlperf
  confidence: high
---

# mlperf-hpc-eval

> MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems — Farrell et al. (2021) (arXiv:2110.11466, 2021)

## What this evaluates

Evaluates end-to-end performance of HPC systems for scientific machine learning, focusing on data staging, I/O efficiency, and model convergence under massive dataset constraints. It measures how well systems handle large-scale volumetric and high-resolution image workloads while meeting strict accuracy targets.

## Datasets

- **CosmoFlow** — total ?; splits: train (262144), test (65536); repo https://github.com/mlcommons/hpc
- **DeepCAM** — total ?; splits: train (121266), test (15158); repo https://github.com/mlcommons/hpc

## Metrics

- `MAE` **(primary)** — range: [0, inf)
  - Mean absolute error between predicted and true cosmological parameters across the test set.
- `IOU` **(primary)** — range: [0, 1]
  - Intersection-over-Union between predicted segmentation masks and ground truth masks, averaged across classes.

## Input / output format

**Input**: CosmoFlow: 3D volumetric cubes of size 128^3 with 4 channels. DeepCAM: 16-channel tensors of size 1152x768.

**Output**: CosmoFlow: 4 predicted cosmological parameters. DeepCAM: 1152x768 segmentation masks for 3 classes.

## Scoring recipe

```python
def score_cosmoflow(y_pred, y_true):
    mae = np.mean(np.abs(y_pred - y_true))
    assert mae < 0.124, 'Quality target not met'
    return mae

def score_deepcam(y_pred, y_true):
    intersection = np.sum((y_pred == y_true) & (y_pred > 0), axis=(1,2,3))
    union = np.sum((y_pred > 0) | (y_true > 0), axis=(1,2,3))
    iou = np.mean(intersection / union)
    assert iou > 0.82, 'Quality target not met'
    return iou
```

## Common pitfalls

- I/O bottlenecks often dominate end-to-end runtime, making storage and interconnect performance more critical than raw compute throughput.
- Runs are only valid if the strict quality targets (MAE < 0.124 or IOU > 0.82) are met; otherwise, timing measurements are discarded.
- CosmoFlow requires splitting 512^3 volumes into 128^3 cubes due to memory constraints, which affects data loading and parallelism strategies.

## Evidence (verbatim from paper)

> The target quality is chosen to be mean-absolute-error (MAE) < 0.124 when scaling the batch size and learning rate above the reference configuration. CosmoFlow training exhibited high variability in the number of epochs to converge, which motivated a requirement of 10 training runs to get a reliable measurement of the time to train. ... The target score is the intersection-over-union (IOU) between the predictions and the targets. The scientifically motivated target score is 0.82, which corresponds to a similarity of 82%.

## Citation

```bibtex
@misc{farrell2021mlperf,
  title={MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems},
  author={Farrell et al. (2021)},
  year={2021},
  note={arXiv:2110.11466}
}
```

- arXiv: 2110.11466