Habitat Predictor Eval

---
name: habitat-predictor-eval
description: Probes the ability of a runtime-based predictor to accurately estimate GPU training iteration execution times and cost-normalized throughput across different DNN architectures and GPU generations without requiring full training runs. Use when the user wants to benchmark on ImageNet, WMT'16, LSUN, or asks about evaluating this task. Reports average prediction error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.00527
  bibtex_key: yu2021habitat
  confidence: high
---

# habitat-predictor-eval

> A Runtime-Based Computational Performance Predictor for Deep Neural Network Training — Yu et al. (2021) (arXiv:2102.00527, 2021)

## What this evaluates

Probes the ability of a runtime-based predictor to accurately estimate GPU training iteration execution times and cost-normalized throughput across different DNN architectures and GPU generations without requiring full training runs.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **WMT'16** — total ?; splits: test (-1)
- **LSUN** — total ?; splits: test (-1)

## Metrics

- `average prediction error` **(primary)** — range: percent
  - Calculated as the absolute difference between predicted and actual iteration execution time divided by the actual time, averaged across all GPU pairs and models, then multiplied by 100.

## Input / output format

**Input**: Model architecture, batch size, source GPU hardware specifications, and target GPU hardware specifications.

**Output**: Predicted training iteration execution time (seconds), training throughput (samples/sec), and cost-normalized throughput (samples/dollar).

## Scoring recipe

```python
total_error = 0.0
count = 0
for pred, actual in zip(predictions, ground_truth):
    if actual > 0:
        total_error += abs(pred - actual) / actual
        count += 1
mape = (total_error / count) * 100
return mape
```

## Common pitfalls

- Uses synthetic data sampled from a normal distribution rather than real dataset values, meaning I/O and data loading overheads are excluded from the measured execution time.
- Averages prediction error across all GPU pairs and models, which can obscure high errors on specific operation types or less common GPU architectures.
- Cost-normalized throughput relies on static rental prices from a specific cloud provider and time period, which may not reflect current market rates or on-premise electricity costs.

## Evidence (verbatim from paper)

> The average prediction error across all GPUs and models is 11.8%.

## Citation

```bibtex
@misc{yu2021habitat,
  title={A Runtime-Based Computational Performance Predictor for Deep Neural Network Training},
  author={Yu et al. (2021)},
  year={2021},
  note={arXiv:2102.00527}
}
```

- arXiv: 2102.00527