News Claim Verification Eval

---
name: news-claim-verification-eval
description: Evaluates a model's ability to verify the veracity of real-world news claims by retrieving relevant evidence documents and predicting a multi-class label. It probes the system's capacity for evidence selection, claim decomposition, and fact-checking under black-box LLM constraints. Use when the user wants to benchmark on RAWFC, LIAR-RAW, or asks about evaluating this task. Reports macro-average F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.17283
  bibtex_key: zhang2024reinforcementretrieval
  confidence: high
---

# news-claim-verification-eval

> Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM — Zhang et al. (2024) (arXiv:2404.17283, 2024)

## What this evaluates

Evaluates a model's ability to verify the veracity of real-world news claims by retrieving relevant evidence documents and predicting a multi-class label. It probes the system's capacity for evidence selection, claim decomposition, and fact-checking under black-box LLM constraints.

## Datasets

- **RAWFC** — total 2012; splits: train (1612), valid (200), test (200)
- **LIAR-RAW** — total 12590; splits: train (10065), valid (1274), test (1251)

## Metrics

- `macro-average F1` **(primary)** — range: percent
  - Macro-averaged precision, recall, and F1 score across all veracity classes. F1 is computed as F1 = 2 * (Precision * Recall) / (Precision + Recall). Macro-averaging treats all classes equally regardless of frequency.

## Input / output format

**Input**: News claim text, optionally decomposed into intermediate questions, and a set of retrieved evidence documents (top-K or top-20 from a dense retrieval index).

**Output**: A veracity class label prediction from the dataset's predefined set (e.g., True, False, Half for RAWFC; True, Mostly-true, Half-true, Barely-true, False, Pants-fire for LIAR-RAW).

## Scoring recipe

```python
def compute_macro_f1(y_true, y_pred, classes):
    precisions, recalls, f1s = [], [], []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
        precisions.append(p)
        recalls.append(r)
        f1s.append(f1)
    return sum(precisions) / len(classes), sum(recalls) / len(classes), sum(f1s) / len(classes)
```

## Common pitfalls

- Ground-truth leakage in original datasets can artificially inflate performance; the authors explicitly remove leaked documents before evaluation.
- Using Wikipedia-based datasets (e.g., FEVER) risks data contamination since Wikipedia is heavily present in LLM pretraining corpora.
- Macro-averaging is required due to class imbalance across the six veracity categories in LIAR-RAW.

## Evidence (verbatim from paper)

> Following Yang et al. ([2022b]), we use macro-average precision ($P$), recall ($R$), and $F_{1}$ ($F_{1}\=\frac{2RP}{R+P}$) scores for evaluation. We utilize the supplied division of train-validate-test for both data sets. Specifically, they employed an 8/1/1 split ratio for the train/validation/test for the two datasets. Therefore, the corresponding number of samples in the RAWFC data for train/valid/test is 1,612/200/200, and the number of samples in the LIAR-RAW dataset is 10,065/1,274/1,251.

## Citation

```bibtex
@misc{zhang2024reinforcementretrieval,
  title={Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2404.17283}
}
```

- arXiv: 2404.17283