Ioaa Llm Eval

---
name: ioaa-llm-eval
description: Evaluates LLMs' ability to solve advanced astronomy and astrophysics problems, focusing on geometric/spatial reasoning, physical calculations, and multimodal data analysis. It benchmarks performance against human Olympiad participants using official scoring rubrics. Use when the user wants to benchmark on IOAA (International Olympiad on Astronomy and Astrophysics), or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.05016
  bibtex_key: pinheiro2025llmioaa
  confidence: high
---

# ioaa-llm-eval

> Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA) — Lucas Carrit Delgado Pinheiro et al. (2025) (arXiv:2510.05016, 2025)

## What this evaluates

Evaluates LLMs' ability to solve advanced astronomy and astrophysics problems, focusing on geometric/spatial reasoning, physical calculations, and multimodal data analysis. It benchmarks performance against human Olympiad participants using official scoring rubrics.

## Datasets

- **IOAA (International Olympiad on Astronomy and Astrophysics)** — total ?; splits: test (-1); repo https://github.com/OSU-NLP-Group/LLM-IOAA

## Metrics

- `score` **(primary)** — range: percent
  - Percentage of total available points earned by the model, calculated using official IOAA grading rubrics. Scores are normalized to percentages across theory and data analysis exams.
- `medal_threshold` — range: other
  - Categorical ranking (bronze, silver, gold) based on the model's score relative to the human median score: bronze for 100–130%, silver for 130–160%, and gold for >160% of the median.
- `category_accuracy` — range: percent
  - Percentage score broken down by problem type: Category I (Geometric/Spatial) and Category II (Physics/Mathematics).

## Input / output format

**Input**: Text-based astronomy/astrophysics exam questions, supplemented with images, plots, and data tables for data analysis sections.

**Output**: Model-generated step-by-step solutions, calculations, and final answers to each exam problem.

## Scoring recipe

```python
def compute_score(predictions, rubrics):
    total_points = 0
    earned_points = 0
    for q in predictions:
        max_pts = rubrics[q.id].max_points
        earned_pts = rubrics[q.id].grade(q.answer)
        total_points += max_pts
        earned_points += earned_pts
    return (earned_points / total_points) * 100
```

## Common pitfalls

- Small sample size per difficulty level (e.g., 10 easy, 11 medium questions) causes high variance in category-specific scores.
- Data contamination risk: some past exam questions may overlap with known literature or training data (e.g., a 2022 question based on a 1981 study).
- Models struggle with spatial/geometric reasoning even when problems are described purely in text, indicating a fundamental limitation beyond multimodal input processing.

## Evidence (verbatim from paper)

> All scores are normalized to percentages. Each model is ranked separately with respect to the students.

## Citation

```bibtex
@misc{pinheiro2025llmioaa,
  title={Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)},
  author={Lucas Carrit Delgado Pinheiro et al. (2025)},
  year={2025},
  note={arXiv:2510.05016}
}
```

- arXiv: 2510.05016