Fysics Eval

---
name: fysics-eval
description: Evaluates multimodal large language models' ability to perceive, reason about, and generate physical attributes and laws from images, videos, and audio. It probes causal physical reasoning, material property mapping, and cross-modal consistency rather than superficial pattern matching. Use when the user wants to benchmark on FysicsEval, or asks about evaluating this task. Reports average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.07064
  bibtex_key: han2026exploring
  confidence: high
---

# fysics-eval

> Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine — Minghao Han et al. (2026) (arXiv:2602.07064, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to perceive, reason about, and generate physical attributes and laws from images, videos, and audio. It probes causal physical reasoning, material property mapping, and cross-modal consistency rather than superficial pattern matching.

## Datasets

- **FysicsEval** — total ?; splits: test (-1); repo https://github.com/Fysics-AI/FysicsEval

## Metrics

- `average score` **(primary)** — range: percent
  - Arithmetic mean of accuracy or normalized scores across all benchmark subsets. Reported as a percentage or raw score depending on the subset.

## Input / output format

**Input**: Multimodal inputs (images, videos, or audio) paired with text prompts or questions targeting physical attributes, causal relationships, or material properties.

**Output**: Text-based answers or generated images/videos, depending on the task modality (perception vs. generation).

## Scoring recipe

```python
def compute_average_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models may rely on surface-level visual patterns rather than inferring latent causal physical relationships, leading to poor generalization on unseen physical constraints.
- Optimizing heavily for physical causal chains can interfere with purely abstract mathematical or diagrammatic reasoning, causing performance drops on benchmarks like MathVista.
- Generation models might produce high-fidelity visuals that violate physical laws (e.g., incorrect density or elasticity) if not explicitly constrained during training or evaluation.

## Evidence (verbatim from paper)

> As presented in Table [1], OmniFysics demonstrates exceptional performance on physical perception and reasoning benchmarks, not only significantly outperforming models of comparable scale but also exhibiting strong competitiveness against larger state-of-the-art systems. Overall, it achieves the best performance with a leading average score of 49.97.

## Citation

```bibtex
@misc{han2026exploring,
  title={Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine},
  author={Minghao Han et al. (2026)},
  year={2026},
  note={arXiv:2602.07064}
}
```

- arXiv: 2602.07064