Olympiadbench Eval

---
name: olympiadbench-eval
description: This benchmark evaluates the scientific reasoning capabilities of large multimodal models (LMMs) and large language models (LLMs) on Olympiad-level mathematics and physics problems. It specifically probes bilingual (English and Chinese) text-and-image problem solving, computational correctness, and logical consistency in complex, expert-annotated scenarios. Use when the user wants to benchmark on OlympiadBench, or asks about evaluating this task. Reports micro-average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.14008
  bibtex_key: he2024olympiadbench
  confidence: high
---

# olympiadbench-eval

> OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems — He et al. (2024) (arXiv:2402.14008, 2024)

## What this evaluates

This benchmark evaluates the scientific reasoning capabilities of large multimodal models (LMMs) and large language models (LLMs) on Olympiad-level mathematics and physics problems. It specifically probes bilingual (English and Chinese) text-and-image problem solving, computational correctness, and logical consistency in complex, expert-annotated scenarios.

## Datasets

- **OlympiadBench** — total 8476; splits: test (8476); repo https://github.com/OpenBMB/OlympiadBench

## Metrics

- `micro-average accuracy` **(primary)** — range: percent
  - Calculated as the total number of correctly answered questions divided by the total number of questions across all subsets, reported as a percentage. Automatic scoring is applied to open-ended problems with predefined answer types, while theorem proving problems require manual verification.

## Input / output format

**Input**: Zero-shot prompt template containing the problem text (in English or Chinese), subject category, answer type, and a flag for multiple answers. Multimodal instances include associated images. Prompts explicitly prescribe the expected answer format to facilitate machine scoring.

**Output**: A final answer or solution string in the explicitly prescribed format specified in the prompt template, designed for machine-readable extraction and automatic scoring.

## Scoring recipe

```python
correct_count = 0
total_count = 0
for instance in dataset:
    prediction = model.generate(instance.prompt)
    is_correct = automatic_scoring_pipeline(prediction, instance.gold_answer)
    if is_correct:
        correct_count += 1
    total_count += 1
micro_avg_accuracy = (correct_count / total_count) * 100
```

## Common pitfalls

- Theorem proving problems lack a reliable automatic evaluation method, necessitating manual sampling checks which can introduce subjectivity or sampling bias.
- Some closed-source models return missing responses due to API input length limits or strict formatting requirements, which must be accounted for when calculating averages.
- Models restricted to single-image input will underperform on problems requiring multiple images, skewing modality-specific results if not explicitly flagged.

## Evidence (verbatim from paper)

> We first apply each model to generate answers for questions in OlympiadBench using prompts formed by prompt template, with open-source models running on NVIDIA A800 GPUs. Then, we run the automatic scoring pipeline to judge the correctness of the answers as described in subsection[3.4]. Finally, we calculate the micro-average accuracy as the comparing metric.

## Citation

```bibtex
@misc{he2024olympiadbench,
  title={OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2402.14008}
}
```

- arXiv: 2402.14008