Pri Mo Mo Hpo Eval

---
name: pri-mo-mo-hpo-eval
description: Evaluates multi-objective hyperparameter optimization algorithms on deep learning benchmarks, measuring their ability to find high-quality Pareto fronts of validation error and training cost under varying prior conditions and budget constraints. Use when the user wants to benchmark on Yahpo-Gym & PD1 HPO Benchmarks, or asks about evaluating this task. Reports mean dominated hypervolume.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08371
  bibtex_key: basu2025multiobjective
  confidence: high
---

# pri-mo-mo-hpo-eval

> Multi-objective Hyperparameter Optimization in the Age of Deep Learning — Basu et al. (2025) (arXiv:2511.08371, 2025)

## What this evaluates

Evaluates multi-objective hyperparameter optimization algorithms on deep learning benchmarks, measuring their ability to find high-quality Pareto fronts of validation error and training cost under varying prior conditions and budget constraints.

## Datasets

- **Yahpo-Gym & PD1 HPO Benchmarks** — total ?; splits: test (-1)

## Metrics

- `mean dominated hypervolume` **(primary)** — range: other
  - Computes the hypervolume indicator (volume of objective space dominated by the found Pareto front relative to a fixed reference point) for each random seed, then averages across 25 seeds.
- `relative rankings` — range: other
  - Ranks algorithms by their mean dominated hypervolume at each evaluation budget step, then averages the ranks across all benchmarks and seeds to report anytime and final performance.

## Input / output format

**Input**: Hyperparameter configurations sampled by the optimizer, evaluated on deep learning tasks (image classification, language translation) to yield validation error and training cost.

**Output**: A sequence of evaluated configurations and their corresponding (validation error, training cost) objective pairs, culminating in a final Pareto front.

## Scoring recipe

```python
hv_scores = []
for seed in range(25):
    pareto_front = run_optimizer(benchmark, seed, budget=20)
    hv = compute_dominated_hypervolume(pareto_front, reference_point)
    hv_scores.append(hv)
mean_hv = sum(hv_scores) / len(hv_scores)
# Relative ranking: rank algorithms by mean_hv at each budget step, average across benchmarks/seeds
```

## Common pitfalls

- Budget is defined as '20 equivalent full function evaluations'; cheap approximations/low-fidelity runs count fractionally toward the budget, so miscounting distorts the constraint.
- Hypervolume requires a consistent reference point across all benchmarks and seeds; changing it invalidates cross-benchmark comparisons.
- Prior conditions (good/bad/mixed) drastically affect early performance; evaluating only under 'good' priors overstates robustness and masks recovery behavior.

## Evidence (verbatim from paper)

> We base our multi-objective evaluation on the mean dominated hypervolume across 25 seeds and report relative rankings of the algorithms across budgets. Each optimizer-benchmark-seed combination was run for 20 equivalent full function evaluations, corresponding to typical budgets in practical Deep Learning.

## Citation

```bibtex
@misc{basu2025multiobjective,
  title={Multi-objective Hyperparameter Optimization in the Age of Deep Learning},
  author={Basu et al. (2025)},
  year={2025},
  note={arXiv:2511.08371}
}
```

- arXiv: 2511.08371