Climatecause Eval

---
name: climatecause-eval
description: Evaluates large language models' ability to infer correlation directions between event pairs and identify causal chain structures (membership and node position) from climate science text, including implicit and nested causal relations. Use when the user wants to benchmark on ClimateCause, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14856
  bibtex_key: allein2026climatecause
  confidence: high
---

# climatecause-eval

> ClimateCause: Complex and Implicit Causal Structures in Climate Reports — Allein et al. (2026) (arXiv:2604.14856, 2026)

## What this evaluates

Evaluates large language models' ability to infer correlation directions between event pairs and identify causal chain structures (membership and node position) from climate science text, including implicit and nested causal relations.

## Datasets

- **ClimateCause** — total ?; splits: (unstated); repo https://github.com/laallein/ClimateCause

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: $F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$. Reported as mean and standard deviation across 3 runs per prompt strategy.

## Input / output format

**Input**: Varies by task: (1) CorrI/CorrI+RC: a set of causal pairs $R$, optionally with a statement $s$. (2) CCR: a causal graph with nodes $V$ and edges. (3) CCR+ECI+RC: a statement $s$ and a set of events $V$. Inputs are formatted via zero-shot, few-shot, chain-of-thought, or graph-encoding prompts.

**Output**: Binary or multi-class labels: correlation direction (positive/negative), chain membership (yes/no), or chain position (start/middle/end/none).

## Scoring recipe

```python
def compute_f1(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != p and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t != p and t == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- LLMs frequently over-predict chain membership and positions, confusing the 'none' class with actual chain nodes.
- Providing the original statement (RC) does not significantly improve correlation inference performance despite high recall.
- Graph-oriented prompt definitions may misalign with text-only inputs, causing abstract causal chain reasoning failures.

## Evidence (verbatim from paper)

> Table 4: Mean and standard deviation results for each prompting strategy individually (3 runs) and all strategies together (9 runs; Avg).
| Prompt | Precision | Recall | F1 |

## Citation

```bibtex
@misc{allein2026climatecause,
  title={ClimateCause: Complex and Implicit Causal Structures in Climate Reports},
  author={Allein et al. (2026)},
  year={2026},
  note={arXiv:2604.14856}
}
```

- arXiv: 2604.14856