Chemo Eval

---
name: chemo-eval
description: Evaluates multimodal large language models' ability to solve Olympiad-level theoretical chemistry problems requiring visual perception, chemical reasoning, and structured problem-solving. It specifically probes the visual perception bottleneck in chemistry tasks and tests the effectiveness of multi-agent orchestration and structured visual enhancement. Use when the user wants to benchmark on ChemO, or asks about evaluating this task. Reports normalized rubric-based score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.16205
  bibtex_key: xu2025chemlabs
  confidence: high
---

# chemo-eval

> ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025 — Xu et al. (2025) (arXiv:2511.16205, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to solve Olympiad-level theoretical chemistry problems requiring visual perception, chemical reasoning, and structured problem-solving. It specifically probes the visual perception bottleneck in chemistry tasks and tests the effectiveness of multi-agent orchestration and structured visual enhancement.

## Datasets

- **ChemO** — total 9; splits: test (9)

## Metrics

- `normalized rubric-based score` **(primary)** — range: [0, 100]
  - Raw points from a rubric-based grader (built from official IChO rubrics) are mapped to a global 100-point scale using original problem weights. The total score is the sum of weighted contributions across all 9 problems.
- `LLM-as-a-Judge similarity` — range: [0, 1]
  - An external LLM judge evaluates the semantic alignment between the model response and the reference solution, outputting a score in [0,1]. Higher values indicate stronger agreement in content and reasoning.

## Input / output format

**Input**: AER reformulated problem P_AER containing text and images, optionally augmented with structured visual guidance G (e.g., SMILES extracted by OCSR).

**Output**: Answers for all sub-questions, typically in symbolic or textual formats (e.g., SMILES) that can be automatically graded or compared against reference solutions.

## Scoring recipe

```python
def compute_metrics(predictions, problems, rubrics, reference_solutions):
    total_norm_score = 0.0
    for pred, prob, rubric, ref in zip(predictions, problems, rubrics, reference_solutions):
        raw = rubric_grader(pred, rubric)
        total_norm_score += (raw / prob.original_points) * prob.normalized_max
    similarity = external_llm_judge(predictions, reference_solutions)
    return total_norm_score, similarity
```

## Common pitfalls

- The benchmark uses Assessment-Equivalent Reformulation (AER) to convert visual outputs to machine-readable formats; failing to apply AER makes automatic grading impossible.
- The normalized score is a weighted sum based on original IChO point allocations mapped to a 100-point scale, not a simple average across problems.
- LLM-as-a-Judge similarity is a supplementary metric for semantic alignment and should not be used as the primary performance indicator.

## Evidence (verbatim from paper)

> Evaluation Metrics. Tab. 2 reports two metrics for each problem in the format normalized rubric-based score and LLM-as-a-Judge similarity: Normalized rubric-based score. For each problem, the rubric-based grader returns the total points obtained under the unified deductive framework built from the official IChO rubrics. These raw points are mapped to a global 100-point scale using the original weights. The row Original Points lists the raw allocations for P1-P9, which sum to 385. The row Norm. shows the maximum normalized contribution of each problem on this scale. LLM-as-a-Judge similarity. In parallel, we compute an LLM-as-a-Judge similarity score in [0,1] using an external LLM judge that evaluates the semantic alignment between the model response and the reference solution.

## Citation

```bibtex
@misc{xu2025chemlabs,
  title={ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2511.16205}
}
```

- arXiv: 2511.16205