Model Serving Latency Eval

---
name: model-serving-latency-eval
description: Evaluates the inference latency and end-to-end turn-around time of five machine learning model-serving frameworks across four distinct real-world inference scenarios. It probes how framework specialization (DL-specific vs. general-purpose) and input payload size affect serving performance and stability. Use when the user wants to benchmark on Malware detection, Cryptocoin price forecasting, Image classification, Sentiment analysis, or asks about evaluating this task. Reports average_latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10337
  bibtex_key: derosa2024modelserving
  confidence: high
---

# model-serving-latency-eval

> On the Cost of Model-Serving Frameworks: An Experimental Evaluation — De Rosa et al. (2024) (arXiv:2411.10337, 2024)

## What this evaluates

Evaluates the inference latency and end-to-end turn-around time of five machine learning model-serving frameworks across four distinct real-world inference scenarios. It probes how framework specialization (DL-specific vs. general-purpose) and input payload size affect serving performance and stability.

## Datasets

- **Malware detection** — total ?; splits: test (-1)
- **Cryptocoin price forecasting** — total ?; splits: test (-1)
- **Image classification** — total ?; splits: test (-1)
- **Sentiment analysis** — total ?; splits: test (-1)

## Metrics

- `average_latency` **(primary)** — range: seconds
  - Measured in seconds, reporting both the minimum average latency across payload sizes and the maximum latency at the 99th percentile. Calculated as the mean time difference between request submission and response receipt for each inference call.
- `turn_around_time` — range: seconds
  - End-to-end time from receiving a raw user HTTP request to returning post-processed predictions in human-readable format, including preprocessing and postprocessing overhead.

## Input / output format

**Input**: Raw unprocessed data (APK files, numerical sequences, JPEG images, or text strings) categorized into small, medium, and large/high payloads. These are preprocessed into tensors before being sent to the serving framework API.

**Output**: Post-processed predictions in human-readable format (e.g., classification labels or class probabilities) returned via HTTP response.

## Scoring recipe

```python
latencies = []
for request in requests:
    start = time.time()
    response = framework.predict(preprocessed_input)
    end = time.time()
    latencies.append(end - start)
avg_latency = sum(latencies) / len(latencies)
p99_latency = np.percentile(latencies, 99)
return {'avg_latency': avg_latency, 'p99_latency': p99_latency}
```

## Common pitfalls

- Confusing raw inference latency with end-to-end turn-around time, which includes Flask-based preprocessing and postprocessing overhead.
- Ignoring payload size variations, as latency and stability (especially at the 99th percentile) can shift significantly with medium/large inputs due to outliers.
- Overlooking framework-model alignment bias, as TensorFlow Serving was tested with native TensorFlow models while TorchServe required model conversion, potentially skewing results.

## Evidence (verbatim from paper)

> The minimum average latency (among the two frameworks for the three payloads) was 0.0372s, that is 68.37% lower than the one observed for the three general-purpose frameworks (0.1176s); on the other hand, the maximum average latency was 0.0895s, 72.04% lower than the benchmark general-purpose platforms (0.3201s). TensorFlow Serving showed minimum inference times of, respectively, 0.0215s (small payload), 0.0260s (medium payload) and 0.0242s (high payload), and with maximum latencies (at the 99th percentile) of 0.0431s, 0.0445s, 0.0432s.

## Citation

```bibtex
@misc{derosa2024modelserving,
  title={On the Cost of Model-Serving Frameworks: An Experimental Evaluation},
  author={De Rosa et al. (2024)},
  year={2024},
  note={arXiv:2411.10337}
}
```

- arXiv: 2411.10337