Fast Gshare Eval

---
name: fast-gshare-eval
description: Evaluates the performance of a spatio-temporal GPU sharing architecture for serverless deep learning inference. It measures how well the system manages resource multiplexing, isolation, and auto-scaling under varying workloads and allocation configurations. Use when the user wants to benchmark on MLPerf, or asks about evaluating this task. Reports throughput.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.00558
  bibtex_key: gu2023fastgshare
  confidence: high
---

# fast-gshare-eval

> FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference — Gu et al. (2023) (arXiv:2309.00558, 2023)

## What this evaluates

Evaluates the performance of a spatio-temporal GPU sharing architecture for serverless deep learning inference. It measures how well the system manages resource multiplexing, isolation, and auto-scaling under varying workloads and allocation configurations.

## Datasets

- **MLPerf** — total ?; splits: (unstated)

## Metrics

- `throughput` **(primary)** — range: other
  - Requests per second (req/s) processed by the model under a given load.
- `tail latency` — range: other
  - The maximum or high-percentile latency observed per request.
- `GPU utilization` — range: percent
  - Percentage of GPU compute resources actively used.
- `SM occupancy` — range: percent
  - Percentage of Streaming Multiprocessors (SMs) actively executing threads.
- `memory footprint` — range: other
  - Total GPU device memory (in MB) consumed by the model and its runtime context.

## Input / output format

**Input**: Deep learning inference requests (images, text, audio) generated by Grafana k6 load simulator, targeting specific FaaS functions (ResNet, BERT, RNNT, GNMT, ResNeXt, ViT_hug) deployed on Kubernetes with NVIDIA V100 GPUs.

**Output**: System metrics including throughput (req/s), tail latency (ms), GPU utilization (%), SM occupancy (%), and GPU memory footprint (MB) collected via NVIDIA DCGM-Exporter and nvidia-smi.

## Scoring recipe

```python
def compute_metrics(logs):
    throughput = logs['request_count'] / logs['duration_s']
    tail_latency = max(logs['latency_ms'])
    gpu_util = mean(logs['gpu_util_pct'])
    sm_occupancy = mean(logs['sm_occupancy_pct'])
    mem_footprint = max(logs['gpu_mem_mb'])
    slo_violations = sum(1 for l in logs['latency_ms'] if l > slo_threshold) / len(logs['latency_ms'])
    return throughput, tail_latency, gpu_util, sm_occupancy, mem_footprint, slo_violations
```

## Common pitfalls

- Throughput scales linearly with time quota but saturates with SM partition allocation, so comparing across different spatial allocations requires careful normalization.
- Model sharing introduces a fixed 300MB overhead per model, which can increase memory usage for single-pod deployments compared to no-sharing.
- SLO violations are measured against a specific threshold (e.g., 69ms for ResNet), and auto-scaling must be triggered based on current RPS to maintain <1% violation rate.

## Evidence (verbatim from paper)

> The experiment is designed to evaluate the performance of spatial sharing with 100% time allocation under different spatial sharing configuration: no spatial sharing (racing), 12% partition, and 24% partition. As shown in Figure 10, as the number of pods increases, spatial sharing of 12% and 24% partitions results in significant improvements in both throughput and SM occupancy, while also reducing tail latency accordingly.

## Citation

```bibtex
@misc{gu2023fastgshare,
  title={FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference},
  author={Gu et al. (2023)},
  year={2023},
  note={arXiv:2309.00558}
}
```

- arXiv: 2309.00558