Mlperf Input Pipeline Eval

---
name: mlperf-input-pipeline-eval
description: This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems. Use when the user wants to benchmark on ImageNet, COCO, WMT16, WMT17, or asks about evaluating this task. Reports epoch duration.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2101.12127
  bibtex_key: murray2021tfdata
  confidence: high
---

# mlperf-input-pipeline-eval

> tf.data: A Machine Learning Data Processing Framework — Murray et al. (2021) (arXiv:2101.12127, 2021)

## What this evaluates

This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems.

## Datasets

- **ImageNet** — total ?; splits: train (-1), val (-1), test (-1)
- **COCO** — total ?; splits: train (-1), val (-1), test (-1)
- **WMT16** — total ?; splits: train (-1), val (-1), test (-1)
- **WMT17** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `epoch duration` **(primary)** — range: seconds
  - Wall-clock time in seconds to process one full epoch of input data through the pipeline without model training computation.
- `time to convergence` — range: seconds
  - Wall-clock time in seconds to train the model until it reaches a predefined target accuracy metric.

## Input / output format

**Input**: Raw dataset files (images, text sequences) passed through a declarative tf.data pipeline with transformations like map, interleave, and prefetch.

**Output**: Preprocessed batches of data fed sequentially to the model training loop.

## Scoring recipe

```python
def evaluate_pipeline(pipeline_config, baseline_config, target_accuracy):
    baseline_time = run_pipeline_in_loop(baseline_config, epochs=1)
    optimized_time = run_pipeline_in_loop(pipeline_config, epochs=1)
    speedup = baseline_time / optimized_time
    convergence_time = train_model_until_accuracy(pipeline_config, target=target_accuracy)
    return {
        'epoch_duration': optimized_time,
        'throughput_speedup': speedup,
        'time_to_convergence': convergence_time
    }
```

## Common pitfalls

- Baseline pipelines may still utilize implicit TensorFlow parallelism for user-defined map functions, inflating the baseline performance and reducing measured speedup.
- Cross-system comparisons often conflate input pipeline speed with full training stack performance due to differing hardware, frameworks, and expert tuning levels.
- Auto-tuned configurations match expert-tuned performance but may require longer warm-up periods to converge on optimal buffer sizes and parallelism degrees.

## Evidence (verbatim from paper)

> Figure 7 shows the mean duration of a single epoch, normalized to the epoch duration of the baseline, which does not use any tf.data performance-related features.

## Citation

```bibtex
@misc{murray2021tfdata,
  title={tf.data: A Machine Learning Data Processing Framework},
  author={Murray et al. (2021)},
  year={2021},
  note={arXiv:2101.12127}
}
```

- arXiv: 2101.12127