Hipho Physics Olympiad Eval

---
name: hipho-physics-olympiad-eval
description: This benchmark evaluates multimodal physical reasoning and advanced problem-solving capabilities on international and regional physics Olympiad exams. It probes a model's ability to interpret complex diagrams, data, and text, perform multi-step logical derivations, and produce accurate solutions under strict, official scoring rubrics. Use when the user wants to benchmark on HiPhO, or asks about evaluating this task. Reports exam score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.24855
  bibtex_key: yu2025physicsminions
  confidence: high
---

# hipho-physics-olympiad-eval

> PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System — Fangchen Yu et al. (2025) (arXiv:2509.24855, 2025)

## What this evaluates

This benchmark evaluates multimodal physical reasoning and advanced problem-solving capabilities on international and regional physics Olympiad exams. It probes a model's ability to interpret complex diagrams, data, and text, perform multi-step logical derivations, and produce accurate solutions under strict, official scoring rubrics.

## Datasets

- **HiPhO** — total ?; splits: test (-1)

## Metrics

- `exam score` **(primary)** — range: other
  - Points awarded per exam based on official marking schemes, evaluated at both answer-level and step-level. Scores vary by exam maximum (e.g., 30 for IPhO, 25 for F=MA).
- `Pass@k` — range: other
  - The maximum exam score achieved across k independent inference attempts for a given problem or exam.

## Input / output format

**Input**: Multimodal physics Olympiad problems containing text, illustrations, variable figures, and data figures.

**Output**: Step-by-step solutions and final answers to the physics problems.

## Scoring recipe

```python
def compute_exam_score(predictions, gold, marking_scheme):
    # Evaluate each prediction using official step-level and answer-level rubrics
    step_points = [score_steps(pred, gold, marking_scheme) for pred in predictions]
    answer_points = [score_final_answer(pred, gold) for pred in predictions]
    # Combine according to official marking scheme
    total_points = [s + a for s, a in zip(step_points, answer_points)]
    # Pass@k takes the highest score across k attempts
    return max(total_points)
```

## Common pitfalls

- Evaluation temperature is fixed at 0.6, not sampled freely.
- Scoring uses both step-level and answer-level breakdowns per official marking schemes, not just exact-match final answers.
- Pass@k reports the best score across k attempts, not the average or median performance.

## Evidence (verbatim from paper)

> Pass@k evaluates a model’s best score over $k$ independent attempts by taking the highest-scoring solution per problem.

## Citation

```bibtex
@misc{yu2025physicsminions,
  title={PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System},
  author={Fangchen Yu et al. (2025)},
  year={2025},
  note={arXiv:2509.24855}
}
```

- arXiv: 2509.24855