Retrosynthesis Solve Rate Eval

---
name: retrosynthesis-solve-rate-eval
description: Evaluates an LLM's ability to generate valid chemical synthesis pathways for target molecules given reference routes and iterative feedback. Use when the user wants to benchmark on Pistachio Hard, or asks about evaluating this task. Reports solve rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.15567
  bibtex_key: song2025evaluating
  confidence: high
---

# retrosynthesis-solve-rate-eval

> Evaluating Large Language Models in Scientific Discovery — Song et al. (2025) (arXiv:2512.15567, 2025)

## What this evaluates

Evaluates an LLM's ability to generate valid chemical synthesis pathways for target molecules given reference routes and iterative feedback.

## Datasets

- **Pistachio Hard** — total 100; splits: test (100)

## Metrics

- `solve rate` **(primary)** — range: percent
  - Percentage of target molecules for which the LLM produces a valid synthesis route within 100 queries. Validity requires RDKit-parsable molecules, commercially available precursors, exact-match reaction templates, and correct data format.

## Input / output format

**Input**: Target molecule SMILES, up to 3 reference synthesis routes retrieved via Tanimoto similarity, and iterative feedback on route validity.

**Output**: A proposed synthesis route in a prescribed data format, or a revised route if the first step fails.

## Scoring recipe

```python
def score(predictions, gold):
    valid_count = 0
    for pred in predictions:
        if check_molecule_validity(pred) and check_reaction_validity(pred) and check_route_format(pred):
            valid_count += 1
    return (valid_count / 100) * 100
```

## Common pitfalls

- Models may output routes that violate the prescribed data format, causing automatic rejection despite chemically valid steps.
- Reaction templates not in the database trigger fallback logic that may mask true model capability if not tracked separately.

## Evidence (verbatim from paper)

> Using the Pistachio Hard benchmark set of 100 molecule targets, we evaluated the solve rate using multiple LLMs and compared to common methods spanning MCTS and Retro* search... The evaluation runs the LLM framework on prescribed sets of target molecules and reports the number of molecules that return a valid synthesis route, within 100 LLM queries (i.e. the solve rate).

## Citation

```bibtex
@misc{song2025evaluating,
  title={Evaluating Large Language Models in Scientific Discovery},
  author={Song et al. (2025)},
  year={2025},
  note={arXiv:2512.15567}
}
```

- arXiv: 2512.15567