Docker Dl Performance Eval

---
name: docker-dl-performance-eval
description: Evaluates the performance overhead of Docker containers on deep learning workloads by benchmarking CPU, GPU, I/O, and training speed of representative neural networks (FCN, CNN, RNN) across different frameworks. Use when the user wants to benchmark on MNIST, Cifar10, PTB, or asks about evaluating this task. Reports second per batch.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1711.03386
  bibtex_key: xu2017dockerdlperf
  confidence: high
---

# docker-dl-performance-eval

> Performance Evaluation of Deep Learning Tools in Docker Containers — Xu et al. (2017) (arXiv:1711.03386, 2017)

## What this evaluates

Evaluates the performance overhead of Docker containers on deep learning workloads by benchmarking CPU, GPU, I/O, and training speed of representative neural networks (FCN, CNN, RNN) across different frameworks.

## Datasets

- **MNIST** — total 60000; splits: train (-1)
- **Cifar10** — total ?; splits: train (-1)
- **PTB** — total ?; splits: train (-1)

## Metrics

- `second per batch` **(primary)** — range: seconds
  - The wall-clock time required to process a single training batch through the network. Lower values indicate faster training.
- `GFlops` — range: GFlops
  - Giga floating point operations per second, used for CPU/GPU compute benchmarks (HPL, HPCG, matrix multiplication).
- `I/O latency` — range: milliseconds
  - Time taken for disk read/write operations, measured via dd and ioping tools.

## Input / output format

**Input**: Neural network architecture specifications (layer dimensions, activation functions), dataset samples (images or character sequences), and batch size.

**Output**: Training time per batch in seconds.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    # This benchmark measures wall-clock training time, not prediction accuracy.
    # The metric is directly recorded as the time taken to process one batch.
    return measured_time_per_batch
```

## Common pitfalls

- Overhead is workload-dependent: compute-intensive tasks show negligible overhead, while I/O tasks may show different behavior due to caching.
- Standard Docker cannot access GPUs; the experiments specifically use NVIDIA Docker, which acts as a thin wrapper to load GPU drivers.
- Results are averaged over 20 runs, not reported as single-run values.

## Evidence (verbatim from paper)

> We measure the speed in unit of second per batch. All reported results are the average of 20 runs unless otherwise specified.

## Citation

```bibtex
@misc{xu2017dockerdlperf,
  title={Performance Evaluation of Deep Learning Tools in Docker Containers},
  author={Xu et al. (2017)},
  year={2017},
  note={arXiv:1711.03386}
}
```

- arXiv: 1711.03386