Climaqa Eval

---
name: climaqa-eval
description: Evaluates LLMs on climate science question-answering across multiple formats (multiple-choice, freeform, cloze) and complexity levels (base, reasoning, hypothetical). It probes factual recall, scientific reasoning, and the impact of adaptation techniques like RAG, few-shot prompting, and fine-tuning. Use when the user wants to benchmark on ClimaQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.16701
  bibtex_key: manivannan2024climaqa
  confidence: high
---

# climaqa-eval

> ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models — Manivannan et al. (2024) (arXiv:2410.16701, 2024)

## What this evaluates

Evaluates LLMs on climate science question-answering across multiple formats (multiple-choice, freeform, cloze) and complexity levels (base, reasoning, hypothetical). It probes factual recall, scientific reasoning, and the impact of adaptation techniques like RAG, few-shot prompting, and fine-tuning.

## Datasets

- **ClimaQA** — total 3566; splits: test (3566); repo https://github.com/Rose-STL-Lab/genie-climaqa

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions where the model's top predicted token matches the ground-truth option letter.
- `exact-match (EM)` — range: percent
  - Exact string match between the model's generated single word and the ground-truth answer for cloze questions.
- `partial-score (PS)` — range: [0, 1]
  - Token-level or word-level overlap score for cloze questions, ranging from 0 to 1.
- `BLEU` — range: percent
  - Standard n-gram overlap metric for freeform answers.
- `BERTScore` — range: [0, 1]
  - Contextual embedding-based similarity metric for freeform answers.
- `factual_accuracy` — range: [0, 1]
  - Expert or automated metric designed to measure scientific correctness without bias towards the QA-generation model.

## Input / output format

**Input**: Question text, optionally with multiple-choice options or a blank for cloze. For RAG settings, relevant textbook chunks are appended as context.

**Output**: Single uppercase letter (MCQ), single scientific word (Cloze), or a concise answer of maximum 2 sentences (Freeform).

## Scoring recipe

```python
def score_mcq(pred, gold):
    return 1.0 if pred.strip().upper() == gold.strip().upper() else 0.0
def score_cloze_em(pred, gold):
    return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0
def score_cloze_ps(pred, gold):
    return len(set(pred.lower().split()) & set(gold.lower().split())) / max(len(gold.lower().split()), 1)
def aggregate_accuracy(predictions, golds):
    return sum(score_mcq(p, g) for p, g in zip(predictions, golds)) / len(golds) * 100
```

## Common pitfalls

- BLEU and BERTScore scores are biased towards GPT-3.5-turbo, the model used to generate the synthetic questions, making them less reliable for cross-model comparison.
- RAG performance heavily depends on retrieval source: retrieving from textbooks used to generate questions (source) boosts scores, while retrieving from held-out textbooks often degrades performance due to distracting content.
- Freeform questions often yield higher reasoning scores than MCQs for the same model, indicating that forced-choice formats may constrain complex scientific reasoning.

## Evidence (verbatim from paper)

> Finally, We observe that the BLEU and BERTScore metrics are slightly biased towards the model that was used for QA-generation (gpt-3.5-turbo) while this is not seen in the proposed Factual Accuracy metric [5].

## Citation

```bibtex
@misc{manivannan2024climaqa,
  title={ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models},
  author={Manivannan et al. (2024)},
  year={2024},
  note={arXiv:2410.16701}
}
```

- arXiv: 2410.16701