Sparsetir Eval

---
name: sparsetir-eval
description: Evaluates the runtime performance and speedup of sparse deep learning operators (SpMM, SDDMM) and end-to-end models (GraphSAGE, RGCN, Transformers) on GPU hardware using composable sparse formats and transformations. It probes how format decomposition and modular scheduling primitives improve cache utilization, load balancing, and Tensor Core utilization compared to vendor libraries and existing compilers. Use when the user wants to benchmark on cora, citeseer, pubmed, ppi, ogbn-arxiv, ogbn-proteins, reddit, or asks about evaluating this task. Reports speedup.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.04606
  bibtex_key: ye2022sparsetir
  confidence: high
---

# sparsetir-eval

> SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning — Ye et al. (2022) (arXiv:2207.04606, 2022)

## What this evaluates

Evaluates the runtime performance and speedup of sparse deep learning operators (SpMM, SDDMM) and end-to-end models (GraphSAGE, RGCN, Transformers) on GPU hardware using composable sparse formats and transformations. It probes how format decomposition and modular scheduling primitives improve cache utilization, load balancing, and Tensor Core utilization compared to vendor libraries and existing compilers.

## Datasets

- **cora** — total ?; splits: test (-1)
- **citeseer** — total ?; splits: test (-1)
- **pubmed** — total ?; splits: test (-1)
- **ppi** — total ?; splits: test (-1)
- **ogbn-arxiv** — total ?; splits: test (-1)
- **ogbn-proteins** — total ?; splits: test (-1)
- **reddit** — total ?; splits: test (-1)

## Metrics

- `speedup` **(primary)** — range: ratio
  - Ratio of baseline execution time to SparseTIR execution time (T_baseline / T_sparsetir). Geometric mean is used for aggregation across feature sizes or datasets.

## Input / output format

**Input**: Sparse adjacency matrices (in CSR, BSR, or hyb formats) and dense feature matrices representing graph or attention workloads.

**Output**: Execution time per operator or training epoch, reported as a normalized speedup ratio relative to baseline implementations.

## Scoring recipe

```python
def compute_speedup(baseline_time, sparsetir_time):
    return baseline_time / sparsetir_time

def geometric_mean_speedup(speedups):
    import math
    return math.exp(sum(math.log(s) for s in speedups) / len(speedups))

# Example usage:
# speedups = [compute_speedup(t_base, t_sparse) for t_base, t_sparse in timings]
# final_metric = geometric_mean_speedup(speedups)
```

## Common pitfalls

- Padding zeros in composable formats increases FLOPs but improves runtime due to better scheduling and compile-time load balancing.
- Numerical accuracy must be verified against existing frameworks before trusting speedup gains.
- Results are hardware-specific (V100 vs RTX 3070) and not portable across architectures.

## Evidence (verbatim from paper)

> We measure the geometric mean speedup of different SpMM implementations against cuSPARSE for feature size d∈{32,64,128,256,512}.

## Citation

```bibtex
@misc{ye2022sparsetir,
  title={SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2207.04606}
}
```

- arXiv: 2207.04606