Inference Latency

---
name: inference-latency
description: Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios. Use when the user has predictions and gold and needs to compute inference latency.
metadata:
  skill_kind: metric
  source_arxiv: 1906.03109
  bibtex_key: gupta2019architectural
  confidence: high
---

# inference-latency

> The Architectural Implications of Facebook's DNN-based Personalized Recommendation — Gupta et al. (2019) (arXiv:1906.03109, 2019)

## What this evaluates

Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios.

## Datasets

- **Synthetic Recommendation Models (RMC1-RMC3)** — total ?; splits: test (-1)

## Metrics

- `inference latency` **(primary)** — range: ms
  - Measured as the average time to process a single request or batch under strict SLA constraints, typically reported in milliseconds (ms).

## Input / output format

**Input**: Synthetic recommendation model inputs (feature vectors/embeddings) and model parameters in fp32 format, processed in non-unit batch sizes.

**Output**: Inference predictions (fp32), with evaluation focusing on system-level latency and throughput rather than prediction accuracy.

## Scoring recipe

```python
start = time.perf_counter()
for batch in batches:
    model.forward(batch)
end = time.perf_counter()
latency_ms = (end - start) / len(batches)
throughput_rps = len(batches) / (end - start)
```

## Common pitfalls

- Turbo boost is explicitly disabled for all experiments, which may not reflect real-world data center behavior.
- Experiments use synthetic models and single-worker/single-thread setups, which may not capture multi-worker or production-scale concurrency effects.
- fp32 precision is used throughout, ignoring mixed-precision optimizations common in modern recsys inference.

## Evidence (verbatim from paper)

> Sections V and VI describe the tradeoff between the system and micro-architecture designs, and their impact on inference latency and throughput in the data center. We analyze inference performance using a benchmark which accurately represents the execution flow of production-scale models. The benchmark is implemented in Caffe2 with Intel MKL as a backend library. All experiments are run with a single Caffe2 worker and Intel MKL thread. All data and model parameters are stored in fp32 format.

## Citation

```bibtex
@misc{gupta2019architectural,
  title={The Architectural Implications of Facebook's DNN-based Personalized Recommendation},
  author={Gupta et al. (2019)},
  year={2019},
  note={arXiv:1906.03109}
}
```

- arXiv: 1906.03109