Sparse Gpu Kernel Eval

---
name: sparse-gpu-kernel-eval
description: Evaluates the performance and efficiency of custom sparse GPU kernels for SpMM and SDDMM operations against standard libraries like cuSPARSE on deep learning workloads. It measures computational throughput, memory usage, and end-to-end speedups across various model architectures and batch sizes. Use when the user wants to benchmark on Sparse Matrix Dataset from DNNs, or asks about evaluating this task. Reports Geometric mean speedup.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.10901
  bibtex_key: gale2020sparse
  confidence: high
---

# sparse-gpu-kernel-eval

> Sparse GPU Kernels for Deep Learning — Gale et al. (2020) (arXiv:2006.10901, 2020)

## What this evaluates

Evaluates the performance and efficiency of custom sparse GPU kernels for SpMM and SDDMM operations against standard libraries like cuSPARSE on deep learning workloads. It measures computational throughput, memory usage, and end-to-end speedups across various model architectures and batch sizes.

## Datasets

- **Sparse Matrix Dataset from DNNs** — total 3012; splits: benchmark (3012)

## Metrics

- `Geometric mean speedup` **(primary)** — range: other
  - Geometric mean of the ratio (baseline_time / kernel_time) computed across all benchmark problems.
- `Peak throughput (TFLOPs)` — range: other
  - Measured floating-point operations per second divided by 1e12, reported as peak achieved performance.
- `Bits per dimension` — range: other
  - Standard metric for image generation quality; lower values indicate better reconstruction accuracy.

## Input / output format

**Input**: Sparse matrix in CSR format and a dense matrix, along with specified batch size and precision configuration (single or mixed).

**Output**: Kernel execution time or computed throughput (TFLOPs) on the GPU.

## Scoring recipe

```python
def compute_speedup(baseline_times, kernel_times):
    ratios = [b / k for b, k in zip(baseline_times, kernel_times) if k > 0]
    geometric_mean = exp(mean(log(ratios)))
    return geometric_mean
```

## Common pitfalls

- For convolution benchmarks (ResNet-50), the time for the `im2col` transformation is explicitly excluded from the SpMM/SDDMM benchmark timings.
- When comparing against `cusparseConstrainedGeMM` for SDDMM, the time to explicitly transpose the right-hand operand using cuBLAS is included in the baseline's timing.
- Pre-processing overhead for the ASpT baseline is excluded from the reported benchmark times.

## Evidence (verbatim from paper)

> Across all benchmarks, our SpMM and SDDMM kernels show significant advantages over Nvidia cuSPARSE. For single-precision SpMM, our kernel achieves a geometric mean speedup of 3.58× and reaches 4.29TFLOPs, representing 27.3% of single-precision peak. Our kernel outperforms cuSPARSE on 99.75% of the problems in our dataset.

## Citation

```bibtex
@misc{gale2020sparse,
  title={Sparse GPU Kernels for Deep Learning},
  author={Gale et al. (2020)},
  year={2020},
  note={arXiv:2006.10901}
}
```

- arXiv: 2006.10901