Climatecheck Eval

---
name: climatecheck-eval
description: Tests the ability to retrieve relevant scholarly abstracts for climate change claims from social media and classify the relationship between claims and abstracts as supporting, refuting, or inconclusive. Use when the user wants to benchmark on ClimateCheck, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: high
---

# climatecheck-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Tests the ability to retrieve relevant scholarly abstracts for climate change claims from social media and classify the relationship between claims and abstracts as supporting, refuting, or inconclusive.

## Datasets

- **ClimateCheck** — total 435; splits: train (-1), test (-1); HF `rabuahmad/climatecheck`; repo https://sdproc.org/2025/climatecheck.html

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Proportion of claims where the correct abstract is ranked within the top 10 retrieved results.

## Input / output format

**Input**: Social media claim (atomic tweet-styled statement).

**Output**: Top-10 ranked scholarly abstracts (Subtask I) and relation classification (supports/refutes/not enough information) for retrieved pairs (Subtask II).

## Scoring recipe

```python
retrieved_abstracts = model.retrieve_top_k(claim, k=10)
recall_at_10 = 1 if gold_abstract in retrieved_abstracts else 0
pred_relations = model.classify_relations(claim, retrieved_abstracts)
f1_score = f1_score(gold_relations, pred_relations)
```

## Common pitfalls

- Subtask I evaluates retrieval (Recall@10, Bpref) while Subtask II evaluates classification (F1) on the retrieved set, which can be conflated in leaderboard reporting.
- The dataset is small (435 claims), making aggregate metrics highly sensitive to individual claim difficulty and annotation noise.

## Evidence (verbatim from paper)

> Subtask I of ClimateCheck focused on retrieving the top 10 relevant abstracts per claim, evaluated using Recall@K as well as Binary Preference (Bpref), while subtask II asked participants to classify the relation of each claim-abstract pair they retrieved, evaluated using F1-score with additional scaling based on successful retrieval. The top team achieved a Recall@10 score of 0.66, Bpref of 0.49 (subtask I), and and F1 of 0.73 (subtask II).

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141