Moleculeqa Eval

---
name: moleculeqa-eval
description: Evaluates factual accuracy and reliability in molecular comprehension by testing whether models correctly describe molecular properties, structures, applications, and sources without hallucination or omission. It probes domain-specific knowledge retention and consistency against authoritative chemical corpora. Use when the user wants to benchmark on MoleculeQA, or asks about evaluating this task. Reports factual accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.08192
  bibtex_key: [REDACTED]
  confidence: high
---

# moleculeqa-eval

> MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension — Lu et al. (2024) (arXiv:2403.08192, 2024)

## What this evaluates

Evaluates factual accuracy and reliability in molecular comprehension by testing whether models correctly describe molecular properties, structures, applications, and sources without hallucination or omission. It probes domain-specific knowledge retention and consistency against authoritative chemical corpora.

## Datasets

- **MoleculeQA** — total 61574; splits: train (49993), dev (5795), test (5786); repo https://github.com/IDEA-XL/MoleculeQA

## Metrics

- `factual accuracy` **(primary)** — range: [0, 1]
  - Ratio of correct predictions to the total number of evaluated slots. Domain experts classify generated content as correct (matches ground truth), miss (ground truth aspect missing), or error (factual inconsistency).
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric for lexical similarity between generated and reference text.

## Input / output format

**Input**: Molecular reference description or context, along with a question targeting a specific aspect (Structure, Property, Application, or Source).

**Output**: Generated molecular description or selected answer option (positive/negative).

## Scoring recipe

```python
correct = 0
total = 0
for instance in dataset:
    pred = model.generate(instance.question, instance.context)
    judgment = expert_classify(pred, instance.gold)
    if judgment == 'correct':
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Models may achieve high scores on lexical metrics (BLEU/ROUGE) while failing on factual accuracy due to hallucination or omission.
- Evaluation relies on human expert judgment for factual consistency, which is resource-intensive and may introduce annotator subjectivity.
- Scaffold-based data splitting requires careful implementation to prevent structural leakage between train and test sets.

## Evidence (verbatim from paper)

> We define factual accuracy as the ratio of correct predictions to the total number of slots, serving as an average metric to evaluate the reliability of the generated content.

## Citation

```bibtex
@misc{lu2024moleculeqa,
  title={MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2403.08192}
}
```

- arXiv: 2403.08192