Tf2aif Inference Latency Eval

---
name: tf2aif-inference-latency-eval
description: Evaluates the execution latency and code-generation efficiency of automated, platform-specific AI inference engines across heterogeneous cloud-edge hardware (CPU, GPU, ARM, FPGA, SoC) for image classification tasks. Use when the user wants to benchmark on Image classification dataset (unspecified), or asks about evaluating this task. Reports execution latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.13715
  bibtex_key: leftheriotis2024tf2aif
  confidence: high
---

# tf2aif-inference-latency-eval

> TF2AIF: Facilitating development and deployment of accelerated AI models on the cloud-edge continuum — Leftheriotis et al. (2024) (arXiv:2404.13715, 2024)

## What this evaluates

Evaluates the execution latency and code-generation efficiency of automated, platform-specific AI inference engines across heterogeneous cloud-edge hardware (CPU, GPU, ARM, FPGA, SoC) for image classification tasks.

## Datasets

- **Image classification dataset (unspecified)** — total ?; splits: test (-1)

## Metrics

- `execution latency` **(primary)** — range: milliseconds
  - Average time in milliseconds to process a single inference request over 1000 trials per model-platform variant.
- `generation time` — range: seconds
  - Time in seconds to compose Docker images and convert TensorFlow models to platform-specific inference engines.
- `speedup vs native TensorFlow` — range: ratio
  - Ratio of average native TensorFlow latency to average TF2AIF-generated framework latency for the same model and platform.

## Input / output format

**Input**: Single image per inference request.

**Output**: Inference result and execution timestamp for latency measurement.

## Scoring recipe

```python
latencies = []
for variant in variants:
    for _ in range(1000):
        start = time.time()
        _ = send_request(variant, image)
        end = time.time()
        latencies.append((end - start) * 1000)
    avg_latency = sum(latencies) / len(latencies)
    speedup = native_tf_latency / avg_latency
```

## Common pitfalls

- CPU latency shows high variability due to system noise like context switching, requiring careful benchmarking conditions.
- FPGA (ALVEO) conversion time is significantly higher than other platforms due to Vitis-AI toolchain overhead.
- Native TensorFlow comparison excludes FPGA platforms due to lack of native support, making cross-platform speedup comparisons incomplete for FPGAs.

## Evidence (verbatim from paper)

> For our performance benchmarking, we conducted 1000 requests for each AI-framework-platform model variant, focusing on the execution latency measured in milliseconds.

## Citation

```bibtex
@misc{leftheriotis2024tf2aif,
  title={TF2AIF: Facilitating development and deployment of accelerated AI models on the cloud-edge continuum},
  author={Leftheriotis et al. (2024)},
  year={2024},
  note={arXiv:2404.13715}
}
```

- arXiv: 2404.13715