Deeprecsys Eval

---
name: deeprecsys-eval
description: Evaluates the throughput, tail latency, and power efficiency of a dynamic scheduling system for at-scale neural recommendation inference across various industry models, hardware platforms, and tail-latency constraints. Use when the user wants to benchmark on Industry-representative recommendation models (DLRM-RMC1/2/3, WND, MT-WND, NCF, DIN, DIEN), or asks about evaluating this task. Reports QPS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2001.02772
  bibtex_key: gupta2020deeprecsys
  confidence: high
---

# deeprecsys-eval

> DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference — Gupta et al. (2020) (arXiv:2001.02772, 2020)

## What this evaluates

Evaluates the throughput, tail latency, and power efficiency of a dynamic scheduling system for at-scale neural recommendation inference across various industry models, hardware platforms, and tail-latency constraints.

## Datasets

- **Industry-representative recommendation models (DLRM-RMC1/2/3, WND, MT-WND, NCF, DIN, DIEN)** — total ?; splits: test (-1)

## Metrics

- `QPS` **(primary)** — range: other
  - Queries Per Second, normalized to the baseline static scheduler's QPS at the low tail latency case.
- `Tail Latency (p95, p99)` — range: other
  - 95th and 99th percentile of inference latency, measured in milliseconds.
- `QPS/Watt` — range: other
  - Throughput normalized by power consumption, measuring queries processed per watt of energy.

## Input / output format

**Input**: Recommendation inference queries with varying sizes and arrival rates, processed by neural recommendation models.

**Output**: System throughput (QPS), tail latency percentiles (p95, p99), and power efficiency (QPS/Watt) under different tail latency targets and hardware configurations.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions: list of per-query inference latencies
    # gold: dict with 'time_seconds' and 'power_watts'
    qps = len(predictions) / gold['time_seconds']
    p95 = np.percentile(predictions, 95)
    p99 = np.percentile(predictions, 99)
    qps_per_watt = qps / gold['power_watts']
    return {'QPS': qps, 'p95': p95, 'p99': p99, 'QPS/Watt': qps_per_watt}
```

## Common pitfalls

- Tail latency targets (low, medium, high) significantly impact optimal batch size and hardware offloading strategy.
- Power efficiency (QPS/Watt) does not always improve with GPU offloading; memory-intensive models may see degradation due to GPU power overhead.
- Production traffic distributions (heavier tail) differ from synthetic lognormal distributions, requiring careful batch-size tuning.

## Evidence (verbatim from paper)

> Overall, DeepRecSched-CPU achieves 1.7×, 2.1×, and 2.7× higher QPS across all models for the low, medium, and high tail latency targets, respectively. DeepRecSched-CPU is able to increase the overall system throughput by operating at the optimal batch size configuration. Furthermore, DeepRecSched-GPU increases performance improvement to 4.0×, 5.1×, and 5.8× at the low, medium, and high tail latency targets, respectively.

## Citation

```bibtex
@misc{gupta2020deeprecsys,
  title={DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference},
  author={Gupta et al. (2020)},
  year={2020},
  note={arXiv:2001.02772}
}
```

- arXiv: 2001.02772