Ffdl Platform Overhead

---
name: ffdl-platform-overhead
description: Evaluates the runtime performance overhead and scalability of a containerized deep learning platform (FfDL) compared to bare-metal and specialized hardware, using standard image classification benchmarks. It measures how much throughput degrades when running DL training jobs in a Kubernetes-based multi-tenant environment versus direct execution. Use when the user has predictions and gold and needs to compute performance_overhead_pct.
metadata:
  skill_kind: metric
  source_arxiv: 1909.06526
  bibtex_key: jayaram2019ffdl
  confidence: high
---

# ffdl-platform-overhead

> FfDL : A Flexible Multi-tenant Deep Learning Platform — Jayaram et al. (2019) (arXiv:1909.06526, 2019)

## What this evaluates

Evaluates the runtime performance overhead and scalability of a containerized deep learning platform (FfDL) compared to bare-metal and specialized hardware, using standard image classification benchmarks. It measures how much throughput degrades when running DL training jobs in a Kubernetes-based multi-tenant environment versus direct execution.

## Datasets

- **ImageNet** — total 1300000; splits: train (-1)

## Metrics

- `performance_overhead_pct` **(primary)** — range: percent
  - Percentage decrease in training throughput relative to a bare-metal baseline. Calculated as ((baseline_throughput - platform_throughput) / baseline_throughput) * 100.
- `throughput_images_per_sec` — range: other
  - Number of images processed per second during model training. Measured directly from the training loop iterations and batch size.

## Input / output format

**Input**: DL training job configuration specifying model architecture (VGG-16, Inception V3, ResNet-50), framework (Caffe v1.0, TensorFlow v1.5), GPU type (K80, P100, V100), number of learners, GPUs per learner, and batch size.

**Output**: Throughput in images processed per second, and percentage decrease in performance relative to a bare-metal baseline.

## Scoring recipe

```python
baseline = run_benchmark_baremetal(model, framework, gpu_config)
platform = run_benchmark_ffdl(model, framework, gpu_config)
overhead = ((baseline - platform) / baseline) * 100
return overhead
```

## Common pitfalls

- Overhead varies significantly by GPU type and distributed configuration (learners/GPUs per learner), so reporting a single aggregate number masks significant variance.
- Baseline uses manually managed bare-metal servers rather than a standardized containerized setup, potentially attributing K8s/Docker overhead incorrectly.
- Throughput is highly sensitive to CPU thread allocation and batch size; failing to saturate the GPU will artificially inflate measured overhead.

## Evidence (verbatim from paper)

> Performance is quantified as images processed/sec for training. Caffe v1.0 and TensorFlow v1.5 were used. ... From Table 1, we observe that performance overhead induced by FfDL is minimal (up to ≈ 5%).

## Citation

```bibtex
@misc{jayaram2019ffdl,
  title={FfDL : A Flexible Multi-tenant Deep Learning Platform},
  author={Jayaram et al. (2019)},
  year={2019},
  note={arXiv:1909.06526}
}
```

- arXiv: 1909.06526