Climate Fever Eval

---
name: climate-fever-eval
description: This evaluation probes a model's ability to verify scientific claims in a binary classification setting, specifically testing out-of-domain generalization. It measures performance on Supported vs. Refuted labels, emphasizing robustness when applied to climate-related claims outside the training distribution. Use when the user wants to benchmark on CLIMATE-FEVER, or asks about evaluating this task. Reports Balanced Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11036
  bibtex_key: vinod2026uncertainty
  confidence: high
---

# climate-fever-eval

> Uncertainty-Aware Web-Conditioned Scientific Fact-Checking — Vinod et al. (2026) (arXiv:2604.11036, 2026)

## What this evaluates

This evaluation probes a model's ability to verify scientific claims in a binary classification setting, specifically testing out-of-domain generalization. It measures performance on Supported vs. Refuted labels, emphasizing robustness when applied to climate-related claims outside the training distribution.

## Datasets

- **CLIMATE-FEVER** — total ?; splits: test (-1)

## Metrics

- `Balanced Accuracy` **(primary)** — range: percent
  - Average of recall obtained on each class: (TPR + TNR) / 2. Computed for the Supported and Refuted classes.
- `F1 (Supports)` — range: percent
  - Harmonic mean of precision and recall for the Supported class: 2 * (Precision * Recall) / (Precision + Recall).
- `Recall (Supports)` — range: percent
  - True positive rate for the Supported class: TP / (TP + FN).

## Input / output format

**Input**: Scientific claim paired with context (local evidence snippets or retrieved web documents).

**Output**: A single verification label: 'Supported' or 'Refuted'.

## Scoring recipe

```python
def compute_climate_fever_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Supported')
    fn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Supported')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Refuted')
    tn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Refuted')
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    bal_acc = (tpr + tnr) / 2
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * (prec * tpr) / (prec + tpr) if (prec + tpr) > 0 else 0
    return bal_acc, f1, tpr
```

## Common pitfalls

- Confusing the 2-way (Supported/Refuted) and 3-way (Supported/Refuted/NEI) evaluation regimes, as different datasets use different label sets.
- Reporting standard accuracy instead of Balanced Accuracy, which fails to account for class imbalance in datasets like CLIMATE-FEVER.
- Aggregating per-atom predictions using simple majority vote instead of the calibrated judge, which causes a ~14.6 F1 drop on BIONLI-300.

## Evidence (verbatim from paper)

> Table 2. Main results. Balanced Accuracy, Recall, and F1 on BIONLI-300; Macro-F1 on PubMedFact1k; and Balanced Accuracy, Recall (Supports), and F1 (Supports) on CLIMATE-FEVER, evaluated only on the Supported/Refuted subsets.

## Citation

```bibtex
@misc{vinod2026uncertainty,
  title={Uncertainty-Aware Web-Conditioned Scientific Fact-Checking},
  author={Vinod et al. (2026)},
  year={2026},
  note={arXiv:2604.11036}
}
```

- arXiv: 2604.11036