Crews Ews Budget Eval

---
name: crews-ews-budget-eval
description: Evaluates an AI system's ability to extract and classify budget allocations for Early Warning System (EWS) investments from heterogeneous financial PDF reports. It probes multi-label classification, numerical budget extraction with tolerance, and evidence retrieval/mapping in climate finance contexts. Use when the user wants to benchmark on MDB Evidence Set, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.05104
  bibtex_key: vaghefi2025aiforclimatefinance
  confidence: high
---

# crews-ews-budget-eval

> AI for Climate Finance: Agentic Retrieval and Multi-Step Reasoning for Early Warning System Investments — Vaghefi et al. (2025) (arXiv:2504.05104, 2025)

## What this evaluates

Evaluates an AI system's ability to extract and classify budget allocations for Early Warning System (EWS) investments from heterogeneous financial PDF reports. It probes multi-label classification, numerical budget extraction with tolerance, and evidence retrieval/mapping in climate finance contexts.

## Datasets

- **MDB Evidence Set** — total 500; splits: test (500)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions. A prediction is a true positive only if the pillar label is correct AND the predicted budget amount falls within ±5% of the gold amount relative to the document's total budget ($|\hat{b}_{d,p} - b_{d,p}| \leq 0.05 B_d^{\text{tot}}$).
- `Precision` — range: [0, 1]
  - Ratio of true positives to all positive predictions (TP / (TP + FP)), computed under the same ±5% budget tolerance rule.
- `Recall` — range: [0, 1]
  - Ratio of true positives to all actual positives (TP / (TP + FN)), computed under the same ±5% budget tolerance rule.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall: $2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$.
- `Recall@5` — range: [0, 1]
  - Fraction of gold evidence segments found within the top-5 ranked retrieval results.
- `Total Amount Percentage Error` — range: percent
  - Percentage difference between the sum of predicted pillar amounts ($\hat{B}_d^{\text{tot}}$) and the gold total budget ($B_d^{\text{tot}}$).

## Input / output format

**Input**: Heterogeneous PDF financial reports (CREWS-Fund documents or MDB project documents) containing budget allocations, narrative text, and multi-layout financial tables.

**Output**: Structured JSON output containing predicted budget allocations per EWS pillar (5 pillars), binary pillar indicators, retrieved evidence segments, and evidence-to-pillar mappings.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    tp, fp, fn = 0, 0, 0
    for p in gold['pillars']:
        if gold['labels'][p] == 1:
            if preds['labels'][p] == 1 and abs(preds['budgets'][p] - gold['budgets'][p]) <= 0.05 * gold['total']:
                tp += 1
            else:
                fn += 1
        elif preds['labels'][p] == 1:
            fp += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'precision': prec, 'recall': rec, 'f1': f1}
```

## Common pitfalls

- Ignoring the ±5% budget tolerance rule when calculating TPs; the paper explicitly requires numerical fidelity alongside correct labeling.
- Treating the task as single-label classification instead of multi-label over the five fixed EWS pillars.
- Confusing document-level total budget ($B_d^{\text{tot}}$) with pillar-level amounts when applying the tolerance threshold.

## Evidence (verbatim from paper)

> A prediction for pillar p in document d is counted as a true positive (TP) only if both conditions hold: (a) Correct label. The model assigns the pillar label that is truly present, i.e., y_{d,p} = 1 and \hat{y}_{d,p} = 1. (b) Budget fidelity. The predicted allocation is numerically faithful, i.e., |\hat{b}_{d,p} - b_{d,p}| \leq 0.05 B_d^{\text{tot}}, a ±5% tolerance around the gold amount for that pillar. Table 1 reports the scores: the agent attains 0.87 accuracy, 0.89 precision, 0.83 recall...

## Citation

```bibtex
@misc{vaghefi2025aiforclimatefinance,
  title={AI for Climate Finance: Agentic Retrieval and Multi-Step Reasoning for Early Warning System Investments},
  author={Vaghefi et al. (2025)},
  year={2025},
  note={arXiv:2504.05104}
}
```

- arXiv: 2504.05104