Hipho Eval

---
name: hipho-eval
description: Evaluates multimodal large language models on authentic high school physics Olympiad problems, probing their ability to perform step-level physical reasoning, interpret complex diagrams and data plots, and solve problems across diverse physics subfields under Olympiad-level difficulty. Use when the user wants to benchmark on HiPhO, or asks about evaluating this task. Reports Mean Normalized Score (MNS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.07894
  bibtex_key: yu2025hipho
  confidence: high
---

# hipho-eval

> HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark? — Fangchen Yu et al. (2025) (arXiv:2509.07894, 2025)

## What this evaluates

Evaluates multimodal large language models on authentic high school physics Olympiad problems, probing their ability to perform step-level physical reasoning, interpret complex diagrams and data plots, and solve problems across diverse physics subfields under Olympiad-level difficulty.

## Datasets

- **HiPhO** — total ?; splits: test (-1); repo https://github.com/SciYu/HiPhO

## Metrics

- `Mean Normalized Score (MNS)` **(primary)** — range: percent
  - MNS(M) = (1/N_M) * Σ_{Q∈M} (Exam Score(Q) / Full Mark(Q)) × 100%, where M is a category (modality type or physics field), N_M is the number of questions in M, and Q is a single question.
- `Medal Threshold` — range: other
  - Categorical ranking (Gold/Silver/Bronze) assigned based on total exam score relative to official Olympiad cutoffs. Used to compare model performance against human medalists.

## Input / output format

**Input**: Physics Olympiad exam questions presented in mixed modalities: text-only, text with illustration figures, text with variable-based figures, or text with data figures. Experimental and diagram-generation problems are excluded to ensure fair evaluation.

**Output**: Step-by-step solutions and final numerical answers, graded against official Olympiad marking schemes.

## Scoring recipe

```python
def compute_mns(questions, category):
    scores = []
    for q in questions:
        if q.category == category:
            raw = grade_against_scheme(q.ground_truth, q.model_output)
            norm = (raw / q.full_mark) * 100
            scores.append(norm)
    return sum(scores) / len(scores) if scores else 0

def assign_medal(total_score):
    if total_score >= gold_cutoff: return 'Gold'
    elif total_score >= silver_cutoff: return 'Silver'
    else: return 'Bronze'
```

## Common pitfalls

- Experimental and diagram-generation problems are explicitly excluded from evaluation, meaning the Full Mark for models is lower than for humans.
- Visual complexity significantly impacts scores; models drop sharply on variable-based and data figures compared to text-only or simple illustrations.
- Step-level grading uses official marking schemes rather than simple exact-match on final answers, requiring careful alignment with rubric points.

## Evidence (verbatim from paper)

> To investigate the impact of modality on performance, we categorize all problems into four types: Text-Only (TO), Text+Illustration Figure (TI), Text+Variable Figure (TV), and Text+Data Figure (TD). For each type, we report the Mean Normalized score (MNS), defined as: MNS(M) = (1/N_M)∑_{Q∈M} (Exam Score(Q)/Full Mark(Q))×100%, where M∈{TO,TI,TV,TD}, N_M is the number of questions in M, and Q denotes a single question.

## Citation

```bibtex
@misc{yu2025hipho,
  title={HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?},
  author={Fangchen Yu et al. (2025)},
  year={2025},
  note={arXiv:2509.07894}
}
```

- arXiv: 2509.07894