Physunibench Eval

---
name: physunibench-eval
description: Probes the ability of multimodal large language models to solve undergraduate-level physics problems that require integrating textual descriptions with complex diagrams. It specifically tests multi-step scientific reasoning, mathematical derivation, and conceptual understanding across eight distinct physics sub-disciplines. Use when the user wants to benchmark on PhysUniBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.17667
  bibtex_key: [REDACTED]
  confidence: high
---

# physunibench-eval

> PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models — Wang et al. (2025) (arXiv:2506.17667, 2025)

## What this evaluates

Probes the ability of multimodal large language models to solve undergraduate-level physics problems that require integrating textual descriptions with complex diagrams. It specifically tests multi-step scientific reasoning, mathematical derivation, and conceptual understanding across eight distinct physics sub-disciplines.

## Datasets

- **PhysUniBench** — total 3304; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances. For multiple-choice questions, it uses exact string matching against the correct option. For open-ended questions, it combines symbolic computation (e.g., SymPy) for mathematical equivalence with an LLM judge for reasoning and semantic correctness.

## Input / output format

**Input**: Zero-shot setting: models receive a textual problem description paired with an associated image/diagram.

**Output**: For MC questions: a single selected option. For open-ended questions: the final answer enclosed in LaTeX \boxed{} format.

## Scoring recipe

```python
correct = 0
for pred, gold, q_type in zip(predictions, golds, types):
    if q_type == 'MC':
        if pred.strip() == gold.strip(): correct += 1
    else:
        pred_ans = extract_latex_box(pred)
        if sympy_equivalent(pred_ans, gold): correct += 1
        elif llm_judge_verifies(pred, gold): correct += 1
return (correct / len(predictions)) * 100
```

## Common pitfalls

- Open-ended answers must strictly follow the \boxed{} LaTeX format; otherwise, automated parsing fails.
- The hybrid verification for open-ended questions (SymPy + LLM judge) introduces potential non-determinism and LLM bias not present in pure exact-match metrics.
- Evaluation is strictly zero-shot, so performance heavily depends on the model's inherent reasoning capabilities without in-context examples.

## Evidence (verbatim from paper)

> For MC questions, evaluation is based on exact matching with the correct answer. For OE questions, models must output their final answer using the LaTeX\boxed{} format. Answers are verified through symbolic computation (e.g., SymPy) for mathematical equivalence and a LLM judge (e.g., GPT-4o) for reasoning and semantic correctness. Model performance is reported in terms of accuracy, including overall accuracy across the entire benchmark, as well as accuracy broken down by physics sub-discipline, difficulty level, and question type (open-ended versus multiple-choice).

## Citation

```bibtex
@misc{wang2025physunibench,
  title={PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2506.17667}
}
```

- arXiv: 2506.17667