Climate Eval Eval

---
name: climate-eval-eval
description: This benchmark evaluates open-source large language models on their ability to understand, classify, and reason about climate-related discourse. It probes capabilities across text classification, stance detection, claim verification, misinformation detection, and named entity recognition using real-world news, corporate reports, social media, and scientific abstracts. Use when the user wants to benchmark on Guardian Climate News Corpus, Climate-Stance, Climate-FEVER, Climate-Change NER, Net-Zero Reduction, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18653
  bibtex_key: kurfali2025climateeval
  confidence: high
---

# climate-eval-eval

> Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change — Kurfalı et al. (2025) (arXiv:2505.18653, 2025)

## What this evaluates

This benchmark evaluates open-source large language models on their ability to understand, classify, and reason about climate-related discourse. It probes capabilities across text classification, stance detection, claim verification, misinformation detection, and named entity recognition using real-world news, corporate reports, social media, and scientific abstracts.

## Datasets

- **Guardian Climate News Corpus** — total 40173; splits: test (-1); HF `NLP-RISE/guardian_climate_news_corpus`; repo https://github.com/NLP-RISE/extractguardian
- **Climate-Stance** — total 3777; splits: test (-1)
- **Climate-FEVER** — total 7675; splits: test (-1)
- **Climate-Change NER** — total 534; splits: test (-1)
- **Net-Zero Reduction** — total 3517; splits: test (-1)

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - The unweighted mean of recall (or precision) computed for each class independently, then averaged across all classes. Handles class imbalance by treating all classes equally.

## Input / output format

**Input**: Text input varying by task: article titles, article bodies, or both; claim-evidence pairs; multiple-choice questions with or without context; or raw text passages for stance/sentiment/entity classification.

**Output**: Discrete class label (e.g., topic, stance, sentiment), selected multiple-choice option, or token-level entity spans. Models are prompted in zero-shot or 5-shot settings.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold_labels, classes):
    per_class_f1 = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        per_class_f1.append(f1)
    return sum(per_class_f1) / len(classes)
```

## Common pitfalls

- Task variants exist for the same dataset (e.g., Guardian Title vs Body, SciDCC Title/Summary/Body); evaluators must match the exact variant specified.
- Climate-FEVER is evaluated as a three-way entailment task per claim-evidence pair (7,675 instances), not using the original aggregated general label.
- PIRA tasks use accuracy instead of macro-F1, as noted in the table caption.

## Evidence (verbatim from paper)

> The numbers indicate the models’ performance for each task in F1-macro, except for PIRA (see Section 3.2).

## Citation

```bibtex
@misc{kurfali2025climateeval,
  title={Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change},
  author={Kurfalı et al. (2025)},
  year={2025},
  note={arXiv:2505.18653}
}
```

- arXiv: 2505.18653