Climate Finance Bench Eval

---
name: climate-finance-bench-eval
description: Evaluates Retrieval-Augmented Generation (RAG) systems on climate-finance question answering using expert-validated Q&A pairs from corporate sustainability reports. It measures answer correctness across different retrieval strategies and LLMs, while also quantifying the environmental footprint (GHG emissions) of each configuration. Use when the user wants to benchmark on Climate Finance Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22752
  bibtex_key: mankour2025climatefinancebench
  confidence: high
---

# climate-finance-bench-eval

> Climate Finance Bench — Mankour et al. (2025) (arXiv:2505.22752, 2025)

## What this evaluates

Evaluates Retrieval-Augmented Generation (RAG) systems on climate-finance question answering using expert-validated Q&A pairs from corporate sustainability reports. It measures answer correctness across different retrieval strategies and LLMs, while also quantifying the environmental footprint (GHG emissions) of each configuration.

## Datasets

- **Climate Finance Bench** — total 330; splits: test (330); repo https://github.com/Pladifes/climate_finance_bench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of answers labeled as 'correct' by human annotators or an LLM-as-a-Judge grader. Labels are correct, incomplete, or incorrect, with exactness in numeric values and appropriate textual evidence as key factors.
- `ghg_emissions_per_query` — range: other
  - Estimated greenhouse gas emissions per query in grams of CO2 equivalent, calculated via CodeCarbon for local models and EcoLogits for API models.

## Input / output format

**Input**: A question derived from corporate sustainability reports, paired with retrieved context passages from a specified RAG configuration (minimal or hybrid).

**Output**: A natural language answer generated by the LLM.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, questions, judge="Claude"):
    correct = 0
    for pred, gold, q in zip(predictions, golds, questions):
        label = judge.evaluate(pred, gold, q) # returns "correct", "incomplete", or "incorrect"
        if label == "correct":
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Retrieval quality is the dominant performance bottleneck, not LLM capacity or prompt strategy, so optimizing the retriever yields larger gains than switching to larger models.
- Docling's HTML conversion without post-processing can degrade retrieval performance due to parsing noise, despite preserving layout.
- Emissions estimates for API models have high uncertainty (±4-5 g CO2eq), so comparisons should account for confidence intervals rather than point estimates alone.

## Evidence (verbatim from paper)

> Figure 1: Accuracy breakdown (correct, incomplete, incorrect) for the Minimal RAG configuration across five LLMs.

## Citation

```bibtex
@misc{mankour2025climatefinancebench,
  title={Climate Finance Bench},
  author={Mankour et al. (2025)},
  year={2025},
  note={arXiv:2505.22752}
}
```

- arXiv: 2505.22752