Fpga Inference Latency Throughput

---
name: fpga-inference-latency-throughput
description: Evaluates the end-to-end system performance of an FPGA-accelerated machine learning inference service, specifically measuring inference latency and throughput under varying network conditions and concurrent workloads. Use when the user has predictions and gold and needs to compute round-trip inference latency.
metadata:
  skill_kind: metric
  source_arxiv: 1904.08986
  bibtex_key: duarte2019fpga
  confidence: high
---

# fpga-inference-latency-throughput

> FPGA-accelerated machine learning inference as a service for particle physics computing — Duarte et al. (2019) (arXiv:1904.08986, 2019)

## What this evaluates

Evaluates the end-to-end system performance of an FPGA-accelerated machine learning inference service, specifically measuring inference latency and throughput under varying network conditions and concurrent workloads.

## Datasets

- **Simulated CMS dataset (jet images)** — total ?; splits: (unstated); repo https://github.com/hls-fpga-machine-learning/SonicCMS

## Metrics

- `round-trip inference latency` **(primary)** — range: ms
  - Total time from transmission of the input image to receipt of the classification result, measured in milliseconds. Reported as mean, median, and distribution.
- `throughput` — range: ips
  - Number of inferences completed per second, calculated as total inferences divided by total processing time for concurrent processes.
- `failure_rate` — range: percent
  - Percentage of inference requests that time out during the test.

## Input / output format

**Input**: 2D grayscale image tensor representing a reconstructed particle jet, fed into a ResNet-50 model.

**Output**: Classification result (e.g., jet tag or event type).

## Scoring recipe

```python
start = time.time()
pred = model.predict(image)
latency_ms = (time.time() - start) * 1000
total_time = max(end - start for concurrent jobs)
throughput = (5000 * num_jobs) / total_time
failures = count(requests exceeding timeout)
failure_rate = failures / (5000 * num_jobs)
```

## Common pitfalls

- Network transmission time dominates remote latency (up to 60 ms), while FPGA inference itself is only ~1.8 ms.
- Throughput scales with concurrent CPU processes but plateaus at ~650 ips due to the FPGA pipeline limit, not linearly with process count.
- Remote latency is heavily dependent on physical distance and network conditions, with a hard physical limit of ~10 ms round-trip to Azure East 2.

## Evidence (verbatim from paper)

> We measure the total round-trip latency of the inference request as seen by CMSSW, starting from the transmission of the image and ending with the receipt of the classification results. ... We also measure the throughput based on the total time for each simultaneous process to complete serial processing of 5000 jet images.

## Citation

```bibtex
@misc{duarte2019fpga,
  title={FPGA-accelerated machine learning inference as a service for particle physics computing},
  author={Duarte et al. (2019)},
  year={2019},
  note={arXiv:1904.08986}
}
```

- arXiv: 1904.08986