Molrgen Eval

---
name: molrgen-eval
description: Evaluates large language models' ability to generate novel molecular structures (de novo generation) and predict molecular properties. It probes the models' capacity to optimize for chemical rewards while maintaining structural diversity and validity. Use when the user wants to benchmark on MolRGen, or asks about evaluating this task. Reports top-k score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.18256
  bibtex_key: [REDACTED]
  confidence: high
---

# molrgen-eval

> MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models — Formont et al. (2026) (arXiv:2603.18256, 2026)

## What this evaluates

Evaluates large language models' ability to generate novel molecular structures (de novo generation) and predict molecular properties. It probes the models' capacity to optimize for chemical rewards while maintaining structural diversity and validity.

## Datasets

- **MolRGen** — total 4500; splits: train (-1), test (-1)

## Metrics

- `top-k score` **(primary)** — range: [0, 1]
  - Average reward score of the top-k molecules generated per prompt across $n_r$ sampling attempts. Scores range from 0 to 1.
- `diversity-aware top-k score` — range: [0, 1]
  - Top-k score adjusted by a similarity threshold $s_{max}$ based on Tanimoto similarity. Molecules exceeding the threshold are penalized to enforce structural diversity.
- `normalized Spearman correlation` — range: [0, 1]
  - Spearman rank correlation between predicted and true molecular properties, normalized to [0, 1] for regression tasks.
- `accuracy` — range: [0, 1]
  - Fraction of correctly predicted molecular properties for classification tasks.

## Input / output format

**Input**: Protein structure or molecular prompt specifying target properties.

**Output**: Molecular representation (e.g., SMILES string) representing a generated candidate molecule.

## Scoring recipe

```python
For each prompt, sample $n_r$ molecules.
Score each molecule using a reward function (property prediction + validity).
Sort scores descending.
For top-k, average the top-k scores.
For diversity-aware, apply Tanimoto similarity threshold $s_{max}$ to filter duplicates before averaging.
For property prediction, compute Spearman correlation (regression) or accuracy (classification) against ground truth.
```

## Common pitfalls

- Reinforcement learning can cause models to collapse to a single high-reward molecule, artificially inflating top-1 scores while destroying diversity.
- General-purpose LLMs often output chemically invalid structures, which should be filtered or penalized before scoring.
- Diversity constraints are highly sensitive to the chosen Tanimoto similarity threshold $s_{max}$.

## Evidence (verbatim from paper)

> We report the average top-k scores among $n_{r}$ generations per prompt. Figure 2 presents the diversity-aware top-k scores of various LLMs on the de novo molecular generation task, evaluated across a range of similarity thresholds $s_{	ext{max}}$.

## Citation

```bibtex
@misc{formont2026molrgen,
  title={MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models},
  author={Formont et al. (2026)},
  year={2026},
  note={arXiv:2603.18256}
}
```

- arXiv: 2603.18256