Saih Cosmo Scalability Eval

---
name: saih-cosmo-scalability-eval
description: Evaluates the scalability and performance trends of scientific AI workloads (3D CNNs) on HPC systems under varying node counts and dataset sizes. It probes how hardware constraints like GPU memory, I/O bandwidth, and network communication affect training efficiency and model convergence. Use when the user wants to benchmark on SAIH-cosmo, or asks about evaluating this task. Reports average_flops.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.03410
  bibtex_key: du2022saih
  confidence: high
---

# saih-cosmo-scalability-eval

> SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems — Jiangsu Du et al. (2022) (arXiv:2212.03410, 2022)

## What this evaluates

Evaluates the scalability and performance trends of scientific AI workloads (3D CNNs) on HPC systems under varying node counts and dataset sizes. It probes how hardware constraints like GPU memory, I/O bandwidth, and network communication affect training efficiency and model convergence.

## Datasets

- **SAIH-cosmo** — total 101088; splits: full (101088)

## Metrics

- `average_flops` **(primary)** — range: Tflops
  - Total floating-point operations computed per GPU divided by the wall-clock execution time for that GPU.
- `arithmetic_intensity` — range: other
  - Ratio of total FLOPs to total memory accesses (Read + Write). Calculated per kernel or aggregate to measure compute-to-memory ratio.
- `training_loss` — range: [0, inf)
  - Scalar loss value computed per epoch during training to assess model convergence and accuracy.

## Input / output format

**Input**: Cosmological dataset samples (101,088 total) partitioned into fractions (1/64 to 1/1). 3D CNN models (Small, Medium, Large) with varying parameter counts. Configurations specify node count (1–128), batch size, and precision (FP32/FP16).

**Output**: Training loss per epoch, per-GPU FLOPs, aggregate FLOPs, memory read/write counts, execution time per epoch, and data loading time percentage.

## Scoring recipe

```python
def compute_metrics(flops_per_gpu, exec_time_s, mem_read, mem_write):
    avg_flops = flops_per_gpu / exec_time_s
    arithmetic_intensity = flops_per_gpu / (mem_read + mem_write)
    return avg_flops, arithmetic_intensity
```

## Common pitfalls

- Assuming larger models always yield higher TFLOPs; performance is heavily dictated by arithmetic intensity and memory footprint, not just model size.
- Ignoring I/O bottlenecks during data scaling; PyTorch's data loader can cause severe swapping overhead when datasets exceed node memory capacity.
- Confusing strong scaling (fixed data, varying nodes) with data scaling (fixed nodes, varying data); they probe different bottlenecks (communication vs. I/O).

## Evidence (verbatim from paper)

> The average flops of a single GPU is shown in Fig. 8(a). ... arithmetic intensity of the small model is much lower than those of the other two models. Here 'Read and 'Write' represent the number of memory read accesses and memory write accesses.

## Citation

```bibtex
@misc{du2022saih,
  title={SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems},
  author={Jiangsu Du et al. (2022)},
  year={2022},
  note={arXiv:2212.03410}
}
```

- arXiv: 2212.03410