Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios. Use when the user has predictions and gold and needs to compute inference latency.
Installs into .claude/skills of the current project.
Are you the author of Inference Latency?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-inference-latency)
---
name: inference-latency
description: Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios. Use when the user has predictions and gold and needs to compute inference latency.
metadata:
skill_kind: metric
source_arxiv: 1906.03109
bibtex_key: [REDACTED]
confidence: high
---
# inference-latency
> The Architectural Implications of Facebook's DNN-based Personalized Recommendation — Gupta et al. (2019) (arXiv:1906.03109, 2019)
## What this evaluates
Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios.
## Datasets
- **Synthetic Recommendation Models (RMC1-RMC3)** — total ?; splits: test (-1)
## Metrics
- `inference latency` **(primary)** — range: ms
- Measured as the average time to process a single request or batch under strict SLA constraints, typically reported in milliseconds (ms).
## Input / output format
**Input**: Synthetic recommendation model inputs (feature vectors/embeddings) and model parameters in fp32 format, processed in non-unit batch sizes.
**Output**: Inference predictions (fp32), with evaluation focusing on system-level latency and throughput rather than prediction accuracy.
## Scoring recipe
```python
start = time.perf_counter()
for batch in batches:
model.forward(batch)
end = time.perf_counter()
latency_ms = (end - start) / len(batches)
throughput_rps = len(batches) / (end - start)
```
## Common pitfalls
- Turbo boost is explicitly disabled for all experiments, which may not reflect real-world data center behavior.
- Experiments use synthetic models and single-worker/single-thread setups, which may not capture multi-worker or production-scale concurrency effects.
- fp32 precision is used throughout, ignoring mixed-precision optimizations common in modern recsys inference.
## Evidence (verbatim from paper)
> Sections V and VI describe the tradeoff between the system and micro-architecture designs, and their impact on inference latency and throughput in the data center. We analyze inference performance using a benchmark which accurately represents the execution flow of production-scale models. The benchmark is implemented in Caffe2 with Intel MKL as a backend library. All experiments are run with a single Caffe2 worker and Intel MKL thread. All data and model parameters are stored in fp32 format.
## Citation
```bibtex
@misc{gupta2019architectural,
title={The Architectural Implications of Facebook's DNN-based Personalized Recommendation},
author={Gupta et al. (2019)},
year={2019},
note={arXiv:1906.03109}
}
```
- arXiv: 1906.03109
