Phyx Eval

---
name: phyx-eval
description: Probes multimodal physical reasoning by requiring models to interpret realistic visual scenarios, understand implicit physical conditions, and apply domain-specific knowledge across six physics domains. It evaluates both visual grounding and the ability to integrate symbolic reasoning with real-world constraints. Use when the user wants to benchmark on PhyX, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15929
  bibtex_key: shen2025phyx
  confidence: high
---

# phyx-eval

> PhyX: Does Your Model Have the "Wits" for Physical Reasoning? — Hui Shen et al. (2025) (arXiv:2505.15929, 2025)

## What this evaluates

Probes multimodal physical reasoning by requiring models to interpret realistic visual scenarios, understand implicit physical conditions, and apply domain-specific knowledge across six physics domains. It evaluates both visual grounding and the ability to integrate symbolic reasoning with real-world constraints.

## Datasets

- **PhyX** — total 18000; splits: testmini (6000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances out of the total evaluated. Calculated as (number of correct predictions / total instances) * 100.

## Input / output format

**Input**: An image depicting a realistic physical scenario, accompanied by a problem description (text) and a question. Text input varies across three settings: Full-Text, Text-DeRedundancy, and Text-Minimal.

**Output**: Raw text containing Chain-of-Thought reasoning steps followed by a definitive answer. For multiple-choice questions, the model outputs the option letter or answer text.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers, question_types):
    correct = 0
    for pred, gold, q_type in zip(predictions, gold_answers, question_types):
        # Extract definitive answer from raw CoT output
        extracted = rule_based_extract(pred) if q_type == 'MC' else llm_judge_extract(pred)
        # Compare to ground truth
        if q_type == 'MC':
            is_correct = (extracted == gold) or llm_judge_match(extracted, gold)
        else:
            is_correct = llm_judge_correctness(extracted, gold)
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Open-ended answers can be phrased in myriad ways, making exact string matching ineffective; an LLM judge is required for fair scoring.
- Multiple-choice formats allow models to exploit surface-level cues or eliminate options without genuine reasoning, underestimating true capability gaps.
- Heavy reliance on textual descriptions means performance drops sharply when text is minimized, masking visual reasoning deficits if only full-text prompts are used.

## Evidence (verbatim from paper)

> The quantitative evaluations in all subsequent experiments were assessed on this testmini subset. ... For OE questions, the next step is comparing the extracted answer against the ground truth to determine its correctness. ... For MC questions, we first attempt to directly match the option letter. If this direct matching fails, we then use a LLM as a judge... Notably, even worst human experts achieve accuracy of 75.6%, significantly outperforming all the models included in our comparative analysis.

## Citation

```bibtex
@misc{shen2025phyx,
  title={PhyX: Does Your Model Have the "Wits" for Physical Reasoning?},
  author={Hui Shen et al. (2025)},
  year={2025},
  note={arXiv:2505.15929}
}
```

- arXiv: 2505.15929