Molecule Net Eval

---
name: molecule-net-eval
description: Evaluates a model's ability to predict molecular properties from SMILES strings by fine-tuning on 7 classification benchmarks and measuring performance under scaffold splitting. Use when the user wants to benchmark on MoleculeNet, or asks about evaluating this task. Reports ROC-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.05569
  bibtex_key: zheng2024smieditor
  confidence: high
---

# molecule-net-eval

> SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision — Zheng et al. (2024) (arXiv:2412.05569, 2024)

## What this evaluates

Evaluates a model's ability to predict molecular properties from SMILES strings by fine-tuning on 7 classification benchmarks and measuring performance under scaffold splitting.

## Datasets

- **MoleculeNet** — total ?; splits: test (-1)

## Metrics

- `ROC-AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

## Input / output format

**Input**: Normalized SMILES strings representing molecular structures.

**Output**: Predicted class probabilities or binary/multi-class labels for each molecular property task.

## Scoring recipe

```python
def compute_roc_auc(y_true, y_pred_proba):
    fpr, tpr, _ = roc_curve(y_true, y_pred_proba)
    auc_score = auc(fpr, tpr)
    return auc_score
```

## Common pitfalls

- Uses scaffold splitting rather than random splitting, which is stricter and tests generalization to unseen chemical scaffolds.
- Evaluates each of the 7 tasks separately rather than aggregating scores across tasks before computing the mean.
- Requires normalized SMILES inputs, which must be generated using a specific regular expression tokenizer.

## Evidence (verbatim from paper)

> We evaluate SMI-EDITOR on the MoleculeNet (Wu et al., 2017) benchmark and compare its performance with baseline models. We evaluate SMI-EDITOR on 7 widely-used molecular property prediction tasks ( see Appendix H for details). For all the seven tasks, we take the normalized SMILES information as model input and fine-tuning on each task separately. We use ROC-AUC as the evaluation metric, and the results are summarized in Table 1.

## Citation

```bibtex
@misc{zheng2024smieditor,
  title={SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision},
  author={Zheng et al. (2024)},
  year={2024},
  note={arXiv:2412.05569}
}
```

- arXiv: 2412.05569