Inference Framework Benchmark Eval

---
name: inference-framework-benchmark-eval
description: Evaluates the inference performance of four deep learning frameworks (TensorRT, ONNX Runtime, OpenVINO, TensorFlow XLA) across four CNN architectures on GPU hardware. It probes how configuration settings, graph optimizations, and batch sizes impact inference speed and resource utilization, including co-localized model ensembles. Use when the user wants to benchmark on ImageNet, or asks about evaluating this task. Reports speed.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.04323
  bibtex_key: pocheleu2022inferenceframeworkbenchmark
  confidence: high
---

# inference-framework-benchmark-eval

> Deep Learning Inference Frameworks Benchmark — Pochelu (2022) (arXiv:2210.04323, 2022)

## What this evaluates

Evaluates the inference performance of four deep learning frameworks (TensorRT, ONNX Runtime, OpenVINO, TensorFlow XLA) across four CNN architectures on GPU hardware. It probes how configuration settings, graph optimizations, and batch sizes impact inference speed and resource utilization, including co-localized model ensembles.

## Datasets

- **ImageNet** — total ?; splits: val (-1); repo https://github.com/PierrickPochelu/inference_framework_benchmark

## Metrics

- `speed` **(primary)** — range: other
  - Measured as throughput (images per second) or latency (milliseconds per batch) during inference. The paper reports relative speed changes and absolute timing when comparing framework configurations and batch sizes.

## Input / output format

**Input**: Batches of preprocessed images conforming to framework-specific tensor formats (e.g., NCHW for OpenVINO), fed into frozen/optimized computational graphs on a single GPU.

**Output**: Inference predictions (class logits/probabilities) and timing measurements (latency/throughput) per batch or per image.

## Scoring recipe

```python
for framework in [TensorRT, ONNX_RT, OpenVINO, TF_XLA]:
    for model in [VGG19, ResNet50, DenseNet201, EfficientNetB0]:
        apply_framework_optimizations(framework, model)
        measure_initialization_time(model)
        for batch_size in config.batch_sizes:
            run_inference(model, batch_size)
            record_latency_or_throughput()
report_relative_speed_changes()
```

## Common pitfalls

- Framework performance is highly sensitive to configuration settings; using default settings may not reflect peak speed.
- Graph optimizations like XLA significantly increase initialization time (up to 6x), which must be separated from inference latency.
- Multi-model ensemble performance depends heavily on GPU memory reuse and power limits, not just raw compute.

## Evidence (verbatim from paper)

> In all our benchmarks Tensorflow is accelerated with XLA (Accelerated Linear Algebra). Graph optimization settings: Tensorflow XLA: The computing graph is frozen (i.e., all weights are put in “read-only memory”). The optimizer ”optimize_for_inference_lib” is enabled but it does not impact the performance. XLA is enabled on GPU, disabling it reduces speed by 15%. However, enabling XLA multiplies the initialization time by factor 6. And more, we do not observe performance gain to enable it on the CPU so we let it be disabled on the CPU. ONNX-RT (ONNX-runtime): Caching is enabled, disabling it reduces speed by 3%. The maximum graph level optimization is enabled, and using the default optimization settings reduces speed by 8%. OpenVINO: The “NCHW” tensor format (images, channels, height, and width) is enabled meaning all images must be converted in this format. The convolution fusing is enabled, disabling it reduces speed by 9%.

## Citation

```bibtex
@misc{pocheleu2022inferenceframeworkbenchmark,
  title={Deep Learning Inference Frameworks Benchmark},
  author={Pochelu (2022)},
  year={2022},
  note={arXiv:2210.04323}
}
```

- arXiv: 2210.04323