Molecule Optimization Auc Eval

---
name: molecule-optimization-auc-eval
description: Assesses an LLM's capability to iteratively optimize molecular structures for specific biological targets (JNK3, GSK3β) using evolutionary search guided by generative prompts. Use when the user wants to benchmark on ZINC, or asks about evaluating this task. Reports AUC_top-k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.15567
  bibtex_key: song2025evaluating
  confidence: high
---

# molecule-optimization-auc-eval

> Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

## What this evaluates

Assesses an LLM's capability to iteratively optimize molecular structures for specific biological targets (JNK3, GSK3β) using evolutionary search guided by generative prompts.

## Datasets

- **ZINC** — total ?; splits: reference_pool (-1)

## Metrics

- `AUC_top-k` **(primary)** — range: [0, 1]
  - Area under the curve of the average fitness of the top-10 molecules versus the number of oracle calls (capped at 10,000). Captures both optimization quality and sample efficiency.

## Input / output format

**Input**: One or two parent molecules with their fitness scores, sampled from the current population.

**Output**: A new molecule SMILES string generated via mutation or crossover.

## Scoring recipe

```python
def score(predictions, gold):
    fitness_history = []
    for call in range(10000):
        top10_avg = average_fitness(top_k(predictions, k=10))
        fitness_history.append(top10_avg)
    return trapezoid_area(fitness_history)  # AUC_top-k
```

## Common pitfalls

- Decoding temperature mismatches (e.g., GPT-5 forced to temp=1.0 vs others at 0.8) can artificially inflate or deflate AUC scores.
- High duplication rates in generated molecules reduce effective exploration, penalizing AUC even if final top-k quality is strong.

## Evidence (verbatim from paper)

> Methods were compared using the area under the curve of the top-k average objective versus the number of oracle calls (AUC_top-k) with k = 10, which jointly captures optimization quality and sample efficiency.

## Citation

```bibtex
@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}
```

- arXiv: 2512.15567