Superchem Eval

---
name: superchem-eval
description: Evaluates deep chemical reasoning capabilities of LLMs using expert-curated, entity-masked multiple-choice problems. It probes both final-answer accuracy and the fidelity of the reasoning process against expert-annotated solution paths, while also assessing the impact of multimodal inputs on complex chemical problem-solving. Use when the user wants to benchmark on SUPERChem-A11, SUPERChem-release, SUPERChem-holdout, SUPERChem-100, Multimodal-Essential Subset, or asks about evaluating this task. Reports pass@1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.01274
  bibtex_key: zhao2025superchem
  confidence: high
---

# superchem-eval

> SUPERChem: A Multimodal Reasoning Benchmark in Chemistry — Zhao et al. (2025) (arXiv:2512.01274, 2025)

## What this evaluates

Evaluates deep chemical reasoning capabilities of LLMs using expert-curated, entity-masked multiple-choice problems. It probes both final-answer accuracy and the fidelity of the reasoning process against expert-annotated solution paths, while also assessing the impact of multimodal inputs on complex chemical problem-solving.

## Datasets

- **SUPERChem-A11** — total 633; splits: test (633)
- **SUPERChem-release** — total 500; splits: test (500)
- **SUPERChem-holdout** — total 133; splits: test (133)
- **SUPERChem-100** — total 100; splits: test (100)
- **Multimodal-Essential Subset** — total 238; splits: test (238)

## Metrics

- `pass@1 Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions in a single attempt. For frontier models, reported as Mean Reliability averaged over 8 trials; for others, First Trial accuracy.
- `pass@k Accuracy` — range: [0, 1]
  - Proportion of questions correctly answered in at least one of k independent attempts. Measures latent knowledge accessibility rather than single-trial reliability.
- `Reasoning Path Fidelity (RPF)` — range: [0, 1]
  - Weighted percentage of matched reasoning checkpoints between the model's generated chain-of-thought and the expert-annotated solution path. Evaluated via an independent model-assisted framework.

## Input / output format

**Input**: Entity-masked multiple-choice questions accompanied by chemical structure images. For text-only models, images are replaced by hand-authored descriptive text to ensure informational parity.

**Output**: Selected multiple-choice option. For RPF evaluation, the full chain-of-thought/reasoning path generated by the model.

## Scoring recipe

```python
def score_accuracy(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

def score_passk(predictions_k, golds, k):
    correct = 0
    for i, g in enumerate(golds):
        if any(p == g for p in predictions_k[i][:k]):
            correct += 1
    return correct / len(golds)

def score_rpf(model_cots, expert_paths, checkpoints):
    matched_weight = 0
    total_weight = 0
    for cot, path in zip(model_cots, expert_paths):
        for cp in checkpoints:
            total_weight += cp.weight
            if evaluate_checkpoint_alignment(cot, path, cp):
                matched_weight += cp.weight
    return matched_weight / total_weight
```

## Common pitfalls

- Confusing pass@1 (single-trial reliability) with pass@k (latent knowledge access), leading to misinterpretation of model capability.
- Assuming multimodal input universally improves performance; some models suffer accuracy drops due to cognitive load from visual data.
- Equating high final-answer accuracy with genuine chemical understanding; models can achieve high accuracy via heuristic shortcuts with low RPF.

## Evidence (verbatim from paper)

> pass@1 Accuracy: For frontier models, this value represents the Mean Reliability averaged over 8 trials. For other models, it represents the First Trial accuracy.

## Citation

```bibtex
@misc{zhao2025superchem,
  title={SUPERChem: A Multimodal Reasoning Benchmark in Chemistry},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2512.01274}
}
```

- arXiv: 2512.01274