Ml Drift Inference Eval

---
name: ml-drift-inference-eval
description: Measures inference throughput and latency of large generative models across diverse on-device GPU backends. Probes the efficiency of tensor virtualization and runtime shader generation in decoupling logical semantics from physical memory layouts compared to established inference engines. Use when the user wants to benchmark on Stable Diffusion 1.4, Gemma 2B, Gemma2 2B, Llama 3.2 3B, Llama 3.1 8B, or asks about evaluating this task. Reports tokens/s (decode).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00232
  bibtex_key: tang2025scaling
  confidence: high
---

# ml-drift-inference-eval

> Scaling On-Device GPU Inference for Large Generative Models — Tang et al. (2025) (arXiv:2505.00232, 2025)

## What this evaluates

Measures inference throughput and latency of large generative models across diverse on-device GPU backends. Probes the efficiency of tensor virtualization and runtime shader generation in decoupling logical semantics from physical memory layouts compared to established inference engines.

## Datasets

- **Stable Diffusion 1.4** — total ?; splits: test (-1)
- **Gemma 2B** — total ?; splits: test (-1)
- **Gemma2 2B** — total ?; splits: test (-1)
- **Llama 3.2 3B** — total ?; splits: test (-1)
- **Llama 3.1 8B** — total ?; splits: test (-1)

## Metrics

- `tokens/s (prefill)` — range: other
  - Number of prefill tokens (1024) divided by the wall-clock time to process them.
- `tokens/s (decode)` **(primary)** — range: other
  - Number of generated tokens (256) divided by the wall-clock time to generate them.
- `end-to-end latency (s)` — range: other
  - Total wall-clock time to complete 20 sampling iterations for a 512×512 image.

## Input / output format

**Input**: LLMs: 1280-token context (1024 prefill, 256 generation). Diffusion: 512×512 image generation with 20 sampling iterations.

**Output**: LLMs: sequentially generated tokens. Diffusion: final 512×512 image.

## Scoring recipe

```python
def calc_tokens_per_second(tokens, time_seconds):
    return tokens / time_seconds

def calc_end_to_end_latency(iterations, measure_iteration):
    total_time = sum(measure_iteration() for _ in range(iterations))
    return total_time
```

## Common pitfalls

- OpenCL backends cannot access NVIDIA Tensor Cores, making prefill performance comparisons on NVIDIA GPUs invalid.
- Quantization schemes differ across baselines (ML Drift uses per-channel int8/int4, while others often use q4f16 group quantization), affecting memory bandwidth vs compute-bound characteristics.
- FP32 precision is forced on NVIDIA platforms due to OpenCL driver limitations, skewing performance relative to FP16 CUDA baselines.

## Evidence (verbatim from paper)

> Table 2: LLM performance (tokens/s) on Qualcomm and Arm GPUs for Gemma and Llama models. ... ML Drift’s OpenCL backend achieved a 5× to 11× speedup in token prefill compared to the other open-source LLM inference solutions.

## Citation

```bibtex
@misc{tang2025scaling,
  title={Scaling On-Device GPU Inference for Large Generative Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2505.00232}
}
```

- arXiv: 2505.00232