Spectrumbench Eval

---
name: spectrumbench-eval
description: Evaluates multimodal large language models on spectroscopy tasks spanning signal processing, perception, semantic understanding, and molecular generation. It probes the models' ability to align cross-modal data, reason over spectral patterns, and generate accurate chemical structures or spectra. Use when the user wants to benchmark on SpectrumBench, or asks about evaluating this task. Reports accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01188
  bibtex_key: yang2025spectrumworld
  confidence: high
---

# spectrumbench-eval

> SpectrumWorld: Artificial Intelligence Foundation for Spectroscopy — Zhuo Yang et al. (2025) (arXiv:2508.01188, 2025)

## What this evaluates

Evaluates multimodal large language models on spectroscopy tasks spanning signal processing, perception, semantic understanding, and molecular generation. It probes the models' ability to align cross-modal data, reason over spectral patterns, and generate accurate chemical structures or spectra.

## Datasets

- **SpectrumBench** — total ?; splits: test (-1); repo https://github.com/little1d/SpectrumLab

## Metrics

- `accuracy (%)` **(primary)** — range: percent
  - For multiple-choice tasks, correct answers score 1 and incorrect answers score 0. For generation tasks, a dedicated scoring model (GPT-4o) evaluates the output against the ground truth, normalizing the score between 0 and 1. Non-conforming predictions receive a score of 0. The final metric is the average accuracy across all tasks.

## Input / output format

**Input**: Multiple-choice questions with four options for signal/perception/semantic tasks; molecular structures/SMILES for Molecule-to-Spectrum generation; multiple spectral images for Spectrum-to-Molecule generation.

**Output**: Selected option for multiple-choice tasks; generated spectrum or molecule structure for generation tasks. Predictions must strictly conform to the specified output format, otherwise they are scored 0.

## Scoring recipe

```python
def evaluate(predictions, golds, task_types):
    scores = []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task == "multiple_choice":
            scores.append(1.0 if pred == gold else 0.0)
        elif task == "generation":
            if not matches_format(pred):
                scores.append(0.0)
            else:
                scores.append(gpt4o_score(pred, gold))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Generation task predictions that do not strictly follow the required output format are automatically assigned a score of 0.
- Evaluation of generation tasks relies on an external LLM (GPT-4o) for scoring, which may introduce non-determinism or bias compared to exact-match metrics.
- The benchmark explicitly requires models to generate reasoning trajectories to prevent data leakage, making format compliance and reasoning quality critical for scoring.

## Evidence (verbatim from paper)

> For signal-, perception-, and semantic-level tasks, SpectrumBench standardizes them into a multiple-choice question format, with each question having four options. A correct answer is scored as 1, and an incorrect answer is scored as 0. Generation-level tasks usually do not have fixed-form answers. For Molecule-to-Spectrum tasks, the input is a molecule, and the output is a spectrum. For Spectrum-to-Molecule tasks, the input consists of multiple spectral images, and the output is a molecule. We aim to encourage models to generate meaningful reasoning trajectories rather than simply providing a final answer. This approach can help circumvent the issue of data leakage. Therefore, we use an additional MLLM to score the responses following these steps: (1) Model predictions that do not conform to the specified output format for a given question are assigned a score of zero. (2) For predictions meeting the required format, a dedicated scoring model evaluates the model’s output against the answer, assigning a score normalized between 0 and 1. GPT-4o is employed as the scoring model in our experiment. This design standardizes the primary evaluation metric across all tasks in SpectrumBench

## Citation

```bibtex
@misc{yang2025spectrumworld,
  title={SpectrumWorld: Artificial Intelligence Foundation for Spectroscopy},
  author={Zhuo Yang et al. (2025)},
  year={2025},
  note={arXiv:2508.01188}
}
```

- arXiv: 2508.01188