Physgame Eval

---
name: physgame-eval
description: This benchmark evaluates a model's ability to reason about physical laws and detect physical commonsense violations in gameplay videos. It probes spatial, temporal, and meta-information-based physical reasoning through curated multi-choice questions. Use when the user wants to benchmark on PhysGame, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.01800
  bibtex_key: cao2024physgame
  confidence: high
---

# physgame-eval

> PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos — Cao et al. (2024) (arXiv:2412.01800, 2024)

## What this evaluates

This benchmark evaluates a model's ability to reason about physical laws and detect physical commonsense violations in gameplay videos. It probes spatial, temporal, and meta-information-based physical reasoning through curated multi-choice questions.

## Datasets

- **PhysGame** — total ?; splits: test (-1); repo https://github.com/PhysGame/PhysGame

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multi-choice questions. Average accuracy is computed as the mean accuracy across all 12 evaluated physical domains (e.g., friction, gravity, elasticity, reflection).

## Input / output format

**Input**: Video clips from gameplay footage (frame count and extraction config varies per model, following Video-MME official settings) paired with a multi-choice question asking to identify physical commonsense violations or reason about physical properties.

**Output**: A single selected option (e.g., A, B, C, or D) corresponding to the correct answer for the multi-choice question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(gold)
    return (correct / total) * 100 if total > 0 else 0.0

# Average accuracy across domains:
# avg_acc = np.mean([compute_accuracy(preds[d], gold[d]) for d in domains])
```

## Common pitfalls

- Frame extraction and configuration must strictly follow each model's official Video-MME settings rather than using a fixed number of frames.
- The evaluation prompt is not in the main text but located in the supplementary material, which can lead to inconsistent prompting if overlooked.
- Accuracy is reported per physical domain and as an overall average; reporting only the overall average without domain breakdown loses granularity.

## Evidence (verbatim from paper)

> We employ accuracy as the evaluation metric for our curated multi-choice questions. The evaluation results on the PhysGame benchmark are demonstrated in Table [4]. Among all proprietary models, GPT-4o and Gemini-1.5-pro demonstrate the best performance, achieving average accuracy scores of 56.1% and 55.2%, respectively.

## Citation

```bibtex
@misc{cao2024physgame,
  title={PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos},
  author={Cao et al. (2024)},
  year={2024},
  note={arXiv:2412.01800}
}
```

- arXiv: 2412.01800