Seephys Eval

---
name: seephys-eval
description: This benchmark evaluates multimodal LLMs' ability to perform physics reasoning using visual diagrams, text, or both. It probes visual interpretation, diagram-to-reasoning mapping, and the model's reliance on textual shortcuts versus actual visual perception across varying knowledge levels and diagram types. Use when the user wants to benchmark on SeePhys, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19099
  bibtex_key: xiang2025seephys
  confidence: high
---

# seephys-eval

> SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning — Xiang et al. (2025) (arXiv:2505.19099, 2025)

## What this evaluates

This benchmark evaluates multimodal LLMs' ability to perform physics reasoning using visual diagrams, text, or both. It probes visual interpretation, diagram-to-reasoning mapping, and the model's reliance on textual shortcuts versus actual visual perception across varying knowledge levels and diagram types.

## Datasets

- **SeePhys** — total 2000; splits: test (2000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Evaluated via a composite judgment strategy combining template matching and LLM-based extraction/judging (using DeepSeek-V3). SymPy is first used to screen straightforward answers; remaining responses are compared to ground truth using the LLM judge.

## Input / output format

**Input**: Question text paired with physics diagrams (Text+Vision), question text with diagram captions (Text+Caption), question text only (Text Only), or a composite image rendered from question text and diagrams (Vision Only).

**Output**: A reasoning-augmented response containing significant figures, followed by a final answer extracted via template matching and LLM processing.

## Scoring recipe

```python
def score(response, gold):
    extracted = extract_answer_llm_or_template(response)
    if sympy_check(extracted, gold):
        return 1.0
    judge_prompt = f"Compare extracted answer '{extracted}' with ground truth '{gold}'."
    judge_output = llm_call(judge_prompt, model="DeepSeek-V3")
    return 1.0 if judge_output == "correct" else 0.0

accuracy = mean(score(r, g) for r, g in zip(predictions, golds))
```

## Common pitfalls

- Models may achieve high accuracy in Text-Only settings for vision-optional problems by relying on textual shortcuts rather than visual reasoning, masking true multimodal capabilities.
- The composite judgment strategy uses an LLM (DeepSeek-V3) as the judge, which may introduce bias or inconsistency compared to strict template matching, especially for open-ended physics problems.
- Performance varies significantly across diagram types (e.g., Wave Motion, Circuit Diagram) and knowledge levels; reporting only aggregate accuracy masks domain-specific weaknesses.

## Evidence (verbatim from paper)

> We apply accuracy as the metric for this deterministic evaluation. In the experiments in this paper, we use DeepSeek-V3 as the extraction and judge model.

## Citation

```bibtex
@misc{xiang2025seephys,
  title={SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning},
  author={Xiang et al. (2025)},
  year={2025},
  note={arXiv:2505.19099}
}
```

- arXiv: 2505.19099