Sciagent Scientific Reasoning Eval

---
name: sciagent-scientific-reasoning-eval
description: Evaluates a multi-agent LLM system's ability to solve high-level, cross-disciplinary scientific problems at Olympiad and frontier-exam levels. It probes formal reasoning, proof generation, symbolic derivation, and chemical modeling through adaptive routing and self-verification. Use when the user wants to benchmark on IMO 2025, IMC 2025, IPhO 2024, IPhO 2025, CPhO 2025, IChO 2025, HLE, or asks about evaluating this task. Reports Olympiad Scoring.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08151
  bibtex_key: li2025sciagent
  confidence: high
---

# sciagent-scientific-reasoning-eval

> SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning — Li et al. (2025) (arXiv:2511.08151, 2025)

## What this evaluates

Evaluates a multi-agent LLM system's ability to solve high-level, cross-disciplinary scientific problems at Olympiad and frontier-exam levels. It probes formal reasoning, proof generation, symbolic derivation, and chemical modeling through adaptive routing and self-verification.

## Datasets

- **IMO 2025** — total ?; splits: test (-1)
- **IMC 2025** — total ?; splits: test (-1)
- **IPhO 2024** — total ?; splits: test (-1)
- **IPhO 2025** — total ?; splits: test (-1)
- **CPhO 2025** — total ?; splits: test (-1)
- **IChO 2025** — total ?; splits: test (-1)
- **HLE** — total ?; splits: test (-1)

## Metrics

- `Olympiad Scoring` **(primary)** — range: 0-100 points
  - Sum of points awarded across all problem parts based on official competition scoring rubrics. Evaluated via an LLM grader trained on standard answers and scoring criteria, followed by human expert verification to ensure strict adherence to the rubrics.
- `HLE Consistency Score` — range: other
  - Assesses consistency between model outputs and standard solutions for a subset of HLE mathematics and physics problems.

## Input / output format

**Input**: Scientific problem statements from Olympiad competitions or HLE, including domain-specific constraints and required solution formats.

**Output**: Complete problem-solving process including intermediate reasoning steps, symbolic derivations, and final answer.

## Scoring recipe

```python
def score_olympiad(solution, rubric):
    total_score = 0
    for part in rubric.parts:
        score = llm_grader.evaluate(solution, part.criteria)
        total_score += score
    verified_score = human_verify(total_score, rubric)
    return verified_score
```

## Common pitfalls

- AI graders may over-award points if not strictly constrained by the official rubric; human verification is required to prevent score inflation.
- Olympiad problems often have multiple valid solution paths; the evaluation must accept alternative correct derivations, not just match the standard answer verbatim.
- HLE subset selection is not fully specified, making exact replication of the general reasoning benchmark difficult.

## Evidence (verbatim from paper)

> Olympiad Scoring. For all Olympiad benchmarks, performance is measured using the official scoring criteria for each competition. Problems are typically divided into multiple parts (e.g., Part A, B, C), each with a specific point value. The total score for a problem $S_{\text{total}}$ is the sum of the points $S_{i}$ awarded for each part $i$.

## Citation

```bibtex
@misc{li2025sciagent,
  title={SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2511.08151}
}
```

- arXiv: 2511.08151