Fedhpo B Eval

---
name: fedhpo-b-eval
description: Evaluates the effectiveness of federated hyperparameter optimization (FedHPO) methods across diverse FL tasks. It measures how well optimizers can find high-performing hyperparameter configurations under distributed, communication-constrained, and heterogeneous data settings. Use when the user wants to benchmark on FedHPO-B, or asks about evaluating this task. Reports mean rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.03966
  bibtex_key: wang2022fedhpo
  confidence: high
---

# fedhpo-b-eval

> FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization — Wang et al. (2022) (arXiv:2206.03966, 2022)

## What this evaluates

Evaluates the effectiveness of federated hyperparameter optimization (FedHPO) methods across diverse FL tasks. It measures how well optimizers can find high-performing hyperparameter configurations under distributed, communication-constrained, and heterogeneous data settings.

## Datasets

- **FedHPO-B** — total 20; splits: train (-1), val (-1), test (-1); repo https://github.com/alibaba/FederatedScope

## Metrics

- `best-seen validation loss` — range: [0, inf)
  - The minimum validation loss achieved during the HPO search process for a given task and optimizer. Lower values indicate better optimization performance.
- `mean rank` **(primary)** — range: other
  - For each of the 20 FL tasks, optimizers are ranked by their best-seen validation loss (lower is better). The mean rank is computed across all tasks. Lower values indicate better overall performance.

## Input / output format

**Input**: A federated learning task specification including model architecture, data distribution, system parameters (e.g., sample_rate, max rounds), and a configurable hyperparameter search space.

**Output**: A hyperparameter configuration proposed by the optimizer, along with the corresponding validation loss and rank.

## Scoring recipe

```python
def compute_mean_rank(results_per_task):
    # results_per_task: dict mapping task_id -> list of (optimizer_id, best_val_loss)
    ranks_per_task = {}
    for task_id, runs in results_per_task.items():
        sorted_runs = sorted(runs, key=lambda x: x[1])
        for rank, (opt_id, _) in enumerate(sorted_runs, start=1):
            ranks_per_task.setdefault(opt_id, []).append(rank)
    mean_ranks = {opt_id: sum(ranks)/len(ranks) for opt_id, ranks in ranks_per_task.items()}
    return mean_ranks
```

## Common pitfalls

- Failing to standardize the total communication budget (number of rounds) across single-fidelity and multi-fidelity optimizers.
- Ignoring FL-specific system constraints like client sampling rates (sample_rate) when comparing against centralized HPO baselines.
- Not repeating experiments with different random seeds, leading to high variance in rank comparisons.

## Evidence (verbatim from paper)

> The best-seen validation loss is monitored for each optimizer (for multi-fidelity optimizers, higher fidelity results are preferred over lower ones). We sort the optimizers by their best-seen results and compare their mean ranks on these 20 FL tasks.

## Citation

```bibtex
@misc{wang2022fedhpo,
  title={FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2206.03966}
}
```

- arXiv: 2206.03966