Climategpt Eval

---
name: climategpt-eval
description: Evaluates large language models on climate-specific knowledge, reasoning, and fact-verification, alongside general domain benchmarks for commonsense reasoning and world knowledge. It also tests multilingual capability via cascaded machine translation on an Arabic exam dataset. Use when the user wants to benchmark on ClimaBench, Pira 2.0 MCQ, Exeter Misinformation, HellaSwag, PIQA, OpenBookQA, WinoGrande, MMLU, EXAMS (Arabic), or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.09646
  bibtex_key: thulke2024climategpt
  confidence: high
---

# climategpt-eval

> ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change — Thulke et al. (2024) (arXiv:2401.09646, 2024)

## What this evaluates

Evaluates large language models on climate-specific knowledge, reasoning, and fact-verification, alongside general domain benchmarks for commonsense reasoning and world knowledge. It also tests multilingual capability via cascaded machine translation on an Arabic exam dataset.

## Datasets

- **ClimaBench** — total ?; splits: test (-1)
- **Pira 2.0 MCQ** — total ?; splits: test (-1)
- **Exeter Misinformation** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **OpenBookQA** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **EXAMS (Arabic)** — total ?; splits: test (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Accuracy is computed as the fraction of correctly predicted labels or answers. For ClimaBench, a dataset-weighted average of individual task accuracies is reported.

## Input / output format

**Input**: Text classification prompts or multiple-choice questions with concatenated choices. For fact-verification, claim-evidence pairs or claims alone.

**Output**: Predicted class label or selected multiple-choice option.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100
```

## Common pitfalls

- Models are evaluated using log-probability ranking for MCQs rather than generation, which can favor models with better calibration over actual reasoning.
- ClimaBench results are reported as a weighted average, masking per-dataset performance variations.
- Cascaded MT evaluation introduces translation errors that can unfairly penalize models not natively trained in the target language.

## Evidence (verbatim from paper)

> Our primary evaluation format involves text classification and multiple-choice questions (MCQs), structured as log probability ranking tasks. Table 11 shows the 5-shot results of all models on the set of climate-specific benchmarks. We report 10-shot results on HellaSwag and 5-shot on all other benchmarks. EXAMS (Acc [%])

## Citation

```bibtex
@misc{thulke2024climategpt,
  title={ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change},
  author={Thulke et al. (2024)},
  year={2024},
  note={arXiv:2401.09646}
}
```

- arXiv: 2401.09646