Wximpactbench Eval

---
name: wximpactbench-eval
description: Evaluates large language models' ability to understand and classify disruptive weather impacts from historical and modern newspaper articles, and to answer related questions by ranking relevant information. It specifically probes models' capacity to handle climate-related polysemy, extract nuanced societal responses, and correctly identify passages without weather impacts. Use when the user wants to benchmark on WXImpactBench, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.20249
  bibtex_key: [REDACTED]
  confidence: high
---

# wximpactbench-eval

> WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models — Yongan Yu et al. (arXiv:2505.20249, 2025)

## What this evaluates

Evaluates large language models' ability to understand and classify disruptive weather impacts from historical and modern newspaper articles, and to answer related questions by ranking relevant information. It specifically probes models' capacity to handle climate-related polysemy, extract nuanced societal responses, and correctly identify passages without weather impacts.

## Datasets

- **WXImpactBench** — total 350; splits: test (350); repo https://github.com/Michaelyya/WXImpactBench

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Macro-averaged F1-score across the six impact categories, historical/modern articles, and context length settings. Computed as the harmonic mean of precision and recall for each label, then averaged.
- `row-wise accuracy` — range: [0, 1]
  - Strict metric requiring correct classification of all six impact labels for a given article. Formula: (1/N) * sum_{i=1}^{N} prod_{j=1}^{6} I(y_hat_i^j == y_i^j), where I is the indicator function.
- `accuracy` — range: [0, 1]
  - Standard accuracy averaged across the six impact categories, historical and modern articles, and context lengths.
- `Hit@1` — range: [0, 1]
  - Proportion of queries where the top-ranked answer/document matches the ground truth.
- `nDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 5, measuring ranking quality with logarithmic discounting for lower positions.
- `Recall@5` — range: [0, 1]
  - Proportion of relevant documents/answers found within the top 5 ranked results.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, averaging the reciprocal of the rank of the first relevant item for each query.

## Input / output format

**Input**: Historical or modern newspaper articles (text). For the mixed-context version, articles are split into ~250-token segments. Prompts are provided in Appendix C.2 (classification) and C.3 (QA).

**Output**: For classification: a set of 6 binary labels (one per impact category: infrastructural, political, financial, ecological, agricultural, human health). For QA: a ranked list of candidate answers/documents.

## Scoring recipe

```python
def row_wise_accuracy(preds, golds, num_labels=6):
    correct = sum(1 for p, g in zip(preds, golds) if all(pi == gi for pi, gi in zip(p, g)))
    return correct / len(golds)

# For classification: compute F1, accuracy, and row-wise accuracy per category/context, then average.
# For QA: compute Hit@1, nDCG@5, Recall@5, MRR on ranked lists.
# All LLMs run with temperature=0; final scores are averaged over 3 runs.
```

## Common pitfalls

- Decomposing multi-label classification into multiple independent binary calls instead of simultaneous prediction (the benchmark requires a single LLM call for all six labels).
- Assuming mixed-context chunk labels are automatically inherited from the original article; they are independently annotated, creating negative examples (chunks with no labels) that must be handled correctly.
- Ignoring the sliding window mechanism used in the QA ranking task, which segments articles into three chunks and ranks independently before merging, to mitigate long-context noise.

## Evidence (verbatim from paper)

> For multi-label classification task, we use F1-score, accuracy, and row-wise accuracy as evaluation metrics.
The evaluation via F1-score and accuracy are averaged across the six impact categories, historical and modern articles, and the effect of different context lengths.
Compared to the common F1-score and accuracy, the row-wise accuracy is a strict metric that requires more accurate output as the model should correctly classify all six impact labels for a given article, defined as

|  | $\text{Row-wise Acc.}\=\frac{1}{N}\sum_{i\=1}^{N}\prod_{j\=1}^{6}\mathcal{I}\left(\hat{y}_{i}^{j}\=y_{i}^{j}\right)$ |  |
| --- | --- | --- |

where $N$ is the number of samples, $\hat{y}_{i}^{j}$ denotes the predicted label for the $j$-th category in the $i$-th sample, $y_{i}^{j}$ is the corresponding ground-truth label, and $\mathcal{I}(\cdot)$ is the indicator function.

## Citation

```bibtex
@misc{yu2025wximpactbench,
  title={WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models},
  author={Yongan Yu et al.},
  year={2025},
  note={arXiv:2505.20249}
}
```

- arXiv: 2505.20249