Bayesian Optimization Eval

---
name: bayesian-optimization-eval
description: Evaluates the ability of Bayesian optimization methods to efficiently search discrete spaces (molecules, arithmetic expressions) by maximizing or minimizing a black-box objective function over a limited budget of oracle calls. It probes how well a model aligns its latent representation with the objective landscape to guide search. Use when the user wants to benchmark on Guacamol, TDC DRD3, Arithmetic Expression, or asks about evaluating this task. Reports objective value.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.20258
  bibtex_key: lee2023advancing
  confidence: high
---

# bayesian-optimization-eval

> Advancing Bayesian Optimization via Learning Correlated Latent Space — Lee et al. (2023) (arXiv:2310.20258, 2023)

## What this evaluates

Evaluates the ability of Bayesian optimization methods to efficiently search discrete spaces (molecules, arithmetic expressions) by maximizing or minimizing a black-box objective function over a limited budget of oracle calls. It probes how well a model aligns its latent representation with the objective landscape to guide search.

## Datasets

- **Guacamol, TDC DRD3, Arithmetic Expression** — total ?; splits: (unstated)

## Metrics

- `objective value` **(primary)** — range: other
  - The value returned by the black-box function for a generated candidate. Tasks either maximize (e.g., Guacamol MPO) or minimize (e.g., DRD3, arithmetic error) this value. Reported as mean and standard deviation over 3 random seeds across the oracle call budget.
- `Pearson correlation` — range: [-1, 1]
  - Pearson correlation coefficient between latent space distances (||z_i - z_j||_2) and objective value distances (|y_i - y_j|), measured over the top 10^3 points during optimization.

## Input / output format

**Input**: Candidate discrete structures (molecules in SELFIES format or arithmetic expressions) sampled from a latent space, passed through a decoder to be evaluated by a black-box objective function.

**Output**: Sequence of candidate structures generated over a budget of oracle calls, along with their corresponding objective values.

## Scoring recipe

```python
def compute_metric(predictions, gold, task_type):
    # predictions: list of objective values over oracle calls
    # task_type: 'max' or 'min'
    if task_type == 'max':
        return max(predictions)
    else:
        return min(predictions)
```

## Common pitfalls

- Tasks have different optimization directions (maximization vs minimization); failing to align the metric direction will invert results.
- The 'Score' in ablation tables is recorded at a fixed oracle budget (20k), not the final budget, so comparisons must use consistent oracle counts.
- Results are averaged over 3 repetitions; single-run evaluations will show high variance and may misrank methods.

## Evidence (verbatim from paper)

> Figure 2, 3 represent the graphs that depict the number of oracle calls, i.e., the number of the black-box function evaluations, and the corresponding mean and standard deviation of objective value.

## Citation

```bibtex
@misc{lee2023advancing,
  title={Advancing Bayesian Optimization via Learning Correlated Latent Space},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2310.20258}
}
```

- arXiv: 2310.20258