Climabench Eval

---
name: climabench-eval
description: This benchmark evaluates LLM-based agents on autonomous, open-ended climate science problem-solving. It probes the model's ability to perform data-driven modeling, apply physics-aware constraints, and generate scientifically rigorous analysis reports without human intervention. Use when the user wants to benchmark on ClimaBench, or asks about evaluating this task. Reports Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16922
  bibtex_key: wang2026climagent
  confidence: high
---

# climabench-eval

> ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis — Wang et al. (2026) (arXiv:2604.16922, 2026)

## What this evaluates

This benchmark evaluates LLM-based agents on autonomous, open-ended climate science problem-solving. It probes the model's ability to perform data-driven modeling, apply physics-aware constraints, and generate scientifically rigorous analysis reports without human intervention.

## Datasets

- **ClimaBench** — total 320; splits: test (320); repo https://github.com/usail-hkust/ClimAgent

## Metrics

- `Overall` **(primary)** — range: [0, 10]
  - Multi-dimensional evaluation rubric assessing solution quality across AE, SC, PS, RBA, and MR. Scores are averaged into an Overall metric.
- `AE` — range: [0, 10]
  - Accuracy/Effectiveness score assessing task comprehension and solution correctness.
- `MR` — range: [0, 10]
  - Modeling Rigor score evaluating the scientific soundness and constraint reasoning of the generated model.
- `PS` — range: [0, 10]
  - Problem Solving score measuring the agent's ability to construct coherent, solution-aware models.
- `RBA` — range: [0, 10]
  - Robustness/Breadth score evaluating performance across diverse problem types and domains.
- `SC` — range: [0, 10]
  - Scientific Correctness score verifying the rationality and expert-level validity of the solution.

## Input / output format

**Input**: Open-ended climate science problems curated from research papers published between 2000 and 2025.

**Output**: Autonomous scientific models, analysis reports, or solutions generated by the LLM agent.

## Scoring recipe

```python
def evaluate_climabench(solutions, problems):
    scores = {}
    for sol, prob in zip(solutions, problems):
        # Automatic LLM scoring via GPT-4o
        llm_score = gpt4o.evaluate(sol, prob, rubric=["AE", "SC", "PS", "RBA", "MR"])
        scores[prob.id] = llm_score
    # Human expert verification on 100 random samples
    human_scores = [expert.evaluate(sol, prob) for sol, prob in random.sample(zip(solutions, problems), 100)]
    # Aggregate dimensions and compute Overall
    overall = mean([mean(dim_scores) for dim_scores in scores.values()])
    return overall
```

## Common pitfalls

- Data leakage: Problems from 2025 are evaluated separately from 2000–2024 to prevent memorization of pretraining data.
- Subjectivity in scoring: Automatic LLM scoring requires human expert verification to ensure scientific validity and prevent hallucination bias.

## Evidence (verbatim from paper)

> ClimAgent consistently outperforms all ablated variants under five evaluation metrics. Removing PA significantly reduces AE and MR, indicating that deep task comprehension is essential for rigorous formulation. The absence of CM leads to sharp declines in MR and PS, highlighting its critical role in constructing coherent, scientifically sound models.

## Citation

```bibtex
@misc{wang2026climagent,
  title={ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysis},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.16922}
}
```

- arXiv: 2604.16922