Drugplayground Eval

---
name: drugplayground-eval
description: Evaluates LLMs' ability to generate accurate, chemically plausible drug property descriptions and to produce meaningful text embeddings for drug discovery. It probes descriptive accuracy, lexical/structural alignment with ground truth, and embedding similarity for downstream representation tasks. Use when the user wants to benchmark on MolTextNet, or asks about evaluating this task. Reports Normalized Total score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02346
  bibtex_key: [REDACTED]
  confidence: high
---

# drugplayground-eval

> DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery — Liu et al. (2026) (arXiv:2604.02346, 2026)

## What this evaluates

Evaluates LLMs' ability to generate accurate, chemically plausible drug property descriptions and to produce meaningful text embeddings for drug discovery. It probes descriptive accuracy, lexical/structural alignment with ground truth, and embedding similarity for downstream representation tasks.

## Datasets

- **MolTextNet** — total 862; splits: test (862); repo https://github.com/HelloWorldLTY/drugplayground

## Metrics

- `Normalized Total score` **(primary)** — range: [0, 5]
  - Mean of five individually normalized metrics: BERT, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. Each metric is normalized to a 0–1 scale before averaging, yielding a 0–5 scale.
- `Cosine similarity` — range: [-1, 1]
  - Average cosine similarity between embeddings of LLM-generated drug descriptions and ground-truth descriptions from MolTextNet. Higher values indicate better semantic alignment.

## Input / output format

**Input**: Drug identifier/name, paired with a prompt template (standard, chain-of-thought, or meta-cognition) and a temperature setting.

**Output**: Natural language description of the drug's properties, including chemical structure, molecular weight, and pharmacological attributes.

## Scoring recipe

```python
def compute_normalized_total(generated_text, reference_text):
    scores = [bert_score(generated, reference),
              rouge1(generated, reference),
              rouge2(generated, reference),
              rougeL(generated, reference),
              bleu(generated, reference)]
    normalized = [normalize_to_0_1(s) for s in scores]
    return sum(normalized) / len(normalized)
```

## Common pitfalls

- Assuming lower temperature always improves generation quality; optimal temperature is highly model-dependent.
- Assuming Chain-of-Thought prompting improves factual accuracy; it often introduces hallucinations and redundant phrasing that lower lexical alignment.
- Equating structured chemical formula outputs with higher factual accuracy; they may still contain incorrect properties.

## Evidence (verbatim from paper)

> We then compute both schema-driven and structure-driven metrics to quantitatively compare LLM-generated descriptions, produced under different prompt settings, against the ground-truth descriptions. ... compute the average BERT, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU scores, and summarized in Supplementary Figure S1. In addition, we quantify overall performance using a Normalized Total score, defined as the mean of the five individually normalized metrics.

## Citation

```bibtex
@misc{liu2026drugplayground,
  title={DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.02346}
}
```

- arXiv: 2604.02346