Hyperjump Eval

---
name: hyperjump-eval
description: Evaluates the optimization quality and time efficiency of hyperparameter search algorithms by comparing the test error rate of recommended configurations against wall-clock time across neural architecture and traditional ML benchmarks. It measures how quickly each optimizer converges to near-optimal configurations under sequential and parallel deployment settings. Use when the user wants to benchmark on NATS-Bench, LIBSVM Covertype, or asks about evaluating this task. Reports test_error_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.02479
  bibtex_key: mendes2021hyperjump
  confidence: high
---

# hyperjump-eval

> HyperJump: Accelerating HyperBand via Risk Modelling — Mendes et al. (2021) (arXiv:2108.02479, 2021)

## What this evaluates

Evaluates the optimization quality and time efficiency of hyperparameter search algorithms by comparing the test error rate of recommended configurations against wall-clock time across neural architecture and traditional ML benchmarks. It measures how quickly each optimizer converges to near-optimal configurations under sequential and parallel deployment settings.

## Datasets

- **NATS-Bench** — total 15625; splits: test (-1)
- **LIBSVM Covertype** — total ?; splits: train (-1)

## Metrics

- `test_error_rate` **(primary)** — range: [0, 1]
  - 1.0 minus the classification accuracy on the held-out test set.
- `wall_clock_time` — range: seconds
  - Total elapsed seconds from optimization start to configuration recommendation, including training and overhead.

## Input / output format

**Input**: Hyperparameter configuration vectors (e.g., NN topology connections, SVM kernel/gamma/C) and a computational budget (epochs for NATS, training set size for LIBSVM).

**Output**: Recommended hyperparameter configuration(s) and their corresponding test error rate.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, start_time, end_time):
    accuracy = sum(predictions == gold_labels) / len(gold_labels)
    test_error_rate = 1.0 - accuracy
    wall_clock_time = end_time - start_time
    return {'test_error_rate': test_error_rate, 'wall_clock_time': wall_clock_time}
```

## Common pitfalls

- Confusing the computational budget (epochs or training set size) with actual wall-clock time; speed-up comparisons must be based on elapsed time, not budget units.
- Assuming the global optimum is known for all benchmarks; only NATS-Bench has an exhaustive evaluation of all 15,625 configurations, while LIBSVM Covertype's optimum is unknown.
- Comparing parallel-only baselines (e.g., ASHA) against sequential methods without adjusting for worker pool size, which skews wall-clock time comparisons.

## Evidence (verbatim from paper)

> Figure 2 reports the average loss (i.e., the test error rate) and corresponding standard deviation in the shaded areas as a function of the wall clock time (i.e., training and recommendation time).

## Citation

```bibtex
@misc{mendes2021hyperjump,
  title={HyperJump: Accelerating HyperBand via Risk Modelling},
  author={Mendes et al. (2021)},
  year={2021},
  note={arXiv:2108.02479}
}
```

- arXiv: 2108.02479