Dlbricks Benchmark Eval

---
name: dlbricks-benchmark-eval
description: Evaluates the accuracy of a composable benchmark generation framework in estimating deep learning model inference latency on CPUs, and measures the computational speedup achieved by benchmarking unique layer sequences instead of running full end-to-end models. Use when the user wants to benchmark on 50 DL Models, or asks about evaluating this task. Reports Normalized Latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.07967
  bibtex_key: li2019dlbricks
  confidence: high
---

# dlbricks-benchmark-eval

> DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended) — Li et al. (2019) (arXiv:1911.07967, 2019)

## What this evaluates

Evaluates the accuracy of a composable benchmark generation framework in estimating deep learning model inference latency on CPUs, and measures the computational speedup achieved by benchmarking unique layer sequences instead of running full end-to-end models.

## Datasets

- **50 DL Models** — total 50; splits: test (50)

## Metrics

- `Normalized Latency` **(primary)** — range: percent
  - Geometric mean of the ratio between constructed latency (estimated from layer sequences) and actual end-to-end latency. Values closer to 100% indicate higher estimation accuracy.
- `Benchmarking Time Speedup` — range: ratio
  - Sum of all models' actual end-to-end latencies divided by the sum of all generated layer-sequence benchmark latencies. Higher values indicate greater time savings.

## Input / output format

**Input**: Deep learning model architecture (decomposed into layer sequences of varying granularity G) and target CPU hardware configuration.

**Output**: Estimated inference latency (from layer sequences) and actual end-to-end inference latency per model.

## Scoring recipe

```python
# 1. Collect latencies: run each model/benchmark 100 times, take 20% trimmed mean
actual_lat = [trimmed_mean(run(model, n=100)) for model in models]
constructed_lat = [estimate_from_layers(model, G) for model in models]

# 2. Compute Normalized Latency (Accuracy)
norm_lat = [c / a for c, a in zip(constructed_lat, actual_lat)]
accuracy = geometric_mean(norm_lat)

# 3. Compute Benchmarking Time Speedup
speedup = sum(actual_lat) / sum(constructed_lat)
```

## Common pitfalls

- The evaluation uses a 20% trimmed mean over 100 runs rather than a simple average, which significantly affects reported latency values.
- Accuracy and speedup are highly dependent on the benchmark granularity parameter G; lower G yields higher speedup but slightly lower accuracy, creating a system-dependent trade-off.
- Results are reported on specific Amazon EC2 instances (c4/c5 series with Intel Platinum/Xeon CPUs); performance does not generalize to other CPU architectures or cloud providers without re-evaluation.

## Evidence (verbatim from paper)

> Each model or benchmark is run 100 times and the 20% trimmed mean is reported. The benchmarking time speedup is calculated as the sum of the end-to-end latency of all models divided by the sum of the latency of all the generated benchmarks. Figure 11 shows the geometric mean of the normalized latency (constructed vs end-to-end latency) of all the 50 models across systems and benchmark granularities.

## Citation

```bibtex
@misc{li2019dlbricks,
  title={DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended)},
  author={Li et al. (2019)},
  year={2019},
  note={arXiv:1911.07967}
}
```

- arXiv: 1911.07967