Chemcotb Eval

---
name: chemcotb-eval
description: Evaluates large language models' ability to perform stepwise chemical reasoning and molecular structure manipulation. It probes capabilities in molecular understanding, functional group/ring recognition, scaffold extraction, and SMILES-based molecule editing and reaction prediction. Use when the user wants to benchmark on ChemCoTBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21318
  bibtex_key: li2025beyondchemicalqa
  confidence: high
---

# chemcotb-eval

> Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations — Hao Li et al. (2025) (arXiv:2505.21318, 2025)

## What this evaluates

Evaluates large language models' ability to perform stepwise chemical reasoning and molecular structure manipulation. It probes capabilities in molecular understanding, functional group/ring recognition, scaffold extraction, and SMILES-based molecule editing and reaction prediction.

## Datasets

- **ChemCoTBench** — total 1495; splits: test (-1)

## Metrics

- `MAE` — range: other
  - Mean Absolute Error used for counting functional groups and rings.
- `Tanimoto similarity` — range: [0, 1]
  - Measures structural overlap between predicted and reference Murcko scaffolds.
- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted answers for binary, multiple-choice, and ring-system identification tasks.
- `Pass@1` — range: [0, 1]
  - Proportion of editing tasks where the single generated molecule satisfies all structural instructions.
- `Top-1 accuracy` — range: [0, 1]
  - Exact match rate for generated SMILES strings in reaction prediction tasks.
- `FTS` — range: [0, 1]
  - Fingerprint-based similarity using Morgan, MACCS, or RDKit fingerprints to measure structural similarity between generated and reference molecules.

## Input / output format

**Input**: Source SMILES string, task instruction (e.g., optimization target, editing rule, reaction condition), and sometimes reference molecules or multiple-choice options.

**Output**: Edited or generated SMILES string, predicted numerical property value, or selected multiple-choice option.

## Scoring recipe

```python
def score(prediction, gold, task_type):
    if task_type == 'counting':
        return abs(int(prediction) - int(gold))  # MAE
    elif task_type == 'scaffold':
        return tanimoto_similarity(mol_from_smiles(prediction), mol_from_smiles(gold))
    elif task_type == 'editing':
        return 1.0 if passes_chemical_constraints(prediction, gold) else 0.0  # Pass@1
    elif task_type == 'reaction':
        exact = 1.0 if prediction == gold else 0.0  # Top-1
        sim = max(fingerprint_similarity(prediction, gold, fp) for fp in ['morgan', 'maccs', 'rdkit'])  # FTS
        return exact, sim
    else:
        return 1.0 if prediction == gold else 0.0  # accuracy
```

## Common pitfalls

- SMILES equivalence requires canonicalization or graph isomorphism checking rather than simple string comparison.
- FTS similarity scores depend heavily on the chosen fingerprint type (Morgan, MACCS, RDKit) and radius/length parameters.
- Pass@1 for molecule editing implicitly requires valid chemical valency and ring closure checks, not just instruction matching.

## Evidence (verbatim from paper)

> For understanding tasks, functional group (FG) and ring recognition are treated as counting problems, with mean absolute error (MAE) used to measure precision. Scaffold-level understanding includes extracting Murcko scaffolds, evaluated by Tanimoto similarity, and identifying whether complex ring systems are present, evaluated by accuracy. The SMILES equivalence task is formulated as a binary decision problem, determining whether the target and source SMILES represent the same molecule, and is also evaluated using accuracy. For molecule editing, we use Pass@1 to assess whether the edited molecule meets the instructions. Mechanism route selection is framed as a multiple-choice task and evaluated by accuracy. Other reaction tasks are modeled as SMILES generation problems, where evaluation is based on both Top-1 accuracy and fingerprint-based similarity (FTS), using Morgan, MACCS, and RDKit fingerprints to reflect correctness and structural similarity.

## Citation

```bibtex
@misc{li2025beyondchemicalqa,
  title={Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations},
  author={Hao Li et al. (2025)},
  year={2025},
  note={arXiv:2505.21318}
}
```

- arXiv: 2505.21318