Workflow Benchmark Accuracy Eval

---
name: workflow-benchmark-accuracy-eval
description: Evaluates whether automatically generated scientific workflow benchmarks accurately replicate the execution time and performance characteristics of real scientific workflows under varying hardware architectures and external memory loads. Use when the user wants to benchmark on Montage, 1000Genome, or asks about evaluating this task. Reports execution_time_ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.03170
  bibtex_key: coleman2022wfbench
  confidence: high
---

# workflow-benchmark-accuracy-eval

> WfBench: Automated Generation of Scientific Workflow Benchmarks — Coleman et al. (2022) (arXiv:2210.03170, 2022)

## What this evaluates

Evaluates whether automatically generated scientific workflow benchmarks accurately replicate the execution time and performance characteristics of real scientific workflows under varying hardware architectures and external memory loads.

## Datasets

- **Montage** — total ?; splits: (unstated)
- **1000Genome** — total ?; splits: (unstated)

## Metrics

- `execution_time_ratio` **(primary)** — range: other
  - Ratio of benchmark execution time to real workflow execution time ($T_{bench} / T_{real}$). Values close to 1.0 indicate high accuracy.
- `makespan_percentage_difference` — range: percent
  - Percentage difference in total workflow makespan between the generated benchmark and the real workflow: $((T_{bench} - T_{real}) / T_{real}) \times 100$.

## Input / output format

**Input**: Real workflow task executables (e.g., Montage, 1000Genome), compute node hardware specifications (CPU, cores, RAM), and external load conditions (e.g., stress-ng instances).

**Output**: Generated benchmark configuration (JSON), execution times for both the real workflow task and the instantiated benchmark, and the computed ratio/difference.

## Scoring recipe

```python
def compute_accuracy(real_time, bench_time):
    ratio = bench_time / real_time
    pct_diff = ((bench_time - real_time) / real_time) * 100
    return ratio, pct_diff
```

## Common pitfalls

- Confusing the configuration parameter $f$ (fraction of CPU instructions) with the evaluation metric itself.
- Assuming benchmark accuracy generalizes across hardware architectures without empirical validation under varying memory loads.
- Ignoring the impact of rounding $f$ to multiples of 0.1, which can cause significant deviations in concurrent execution scenarios.

## Evidence (verbatim from paper)

> Each plot is for different Montage and 1000Genome tasks and shows the ratio between the benchmark’s execution time and that of the workflow task (vertical axis) vs. $f$ (horizontal axis). Values above (resp. below) 1.0 correspond to cases in which the benchmark execution is longer (resp. shorter) than that of the workflow task.

## Citation

```bibtex
@misc{coleman2022wfbench,
  title={WfBench: Automated Generation of Scientific Workflow Benchmarks},
  author={Coleman et al. (2022)},
  year={2022},
  note={arXiv:2210.03170}
}
```

- arXiv: 2210.03170