Pubhealth Fact Checking Eval

---
name: pubhealth-fact-checking-eval
description: This benchmark evaluates automated fact-checking systems on public health claims, measuring both their ability to predict the veracity of a claim and the quality of the generated explanations. It probes domain-specific reasoning, extractive-abstractive explanation generation, and formal coherence properties like consistency and relevance. Use when the user wants to benchmark on PUBHEALTH, or asks about evaluating this task. Reports macroF1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.09926
  bibtex_key: kotonya2020explainable
  confidence: high
---

# pubhealth-fact-checking-eval

> Explainable Automated Fact-Checking for Public Health Claims — Kotonya et al. (2020) (arXiv:2010.09926, 2020)

## What this evaluates

This benchmark evaluates automated fact-checking systems on public health claims, measuring both their ability to predict the veracity of a claim and the quality of the generated explanations. It probes domain-specific reasoning, extractive-abstractive explanation generation, and formal coherence properties like consistency and relevance.

## Datasets

- **PUBHEALTH** — total 11832; splits: train (9466), val (1183), test (1183); repo https://github.com/neemakot/Health-Fact-Checking

## Metrics

- `macroF1` **(primary)** — range: percent
  - Macro-averaged F1 score across all classes (TRUE/FALSE).
- `ROUGE-1 F1` — range: percent
  - F1 score for unigram overlap between generated and gold explanations.
- `ROUGE-2 F1` — range: percent
  - F1 score for bigram overlap between generated and gold explanations.
- `ROUGE-L F1` — range: percent
  - F1 score for longest common subsequence overlap between generated and gold explanations.
- `Strong Global Coherence` — range: percent
  - Percentage of explanations where every sentence entails the claim.
- `Weak Global Coherence` — range: percent
  - Percentage of explanations where no sentence contradicts the claim.
- `Local Coherence` — range: percent
  - Percentage of explanations where no two sentences contradict each other.

## Input / output format

**Input**: A public health claim text and its associated evidence or article text.

**Output**: A veracity label (TRUE or FALSE) and a generated explanation text.

## Scoring recipe

```python
def score(predictions, golds, gold_explanations):
    preds = [1 if p == 'TRUE' else 0 for p in predictions]
    golds_bin = [1 if g == 'TRUE' else 0 for g in golds]
    macro_f1 = f1_score(golds_bin, preds, average='macro') * 100
    r1 = rouge_f1(predictions, gold_explanations, n=1)
    r2 = rouge_f1(predictions, gold_explanations, n=2)
    rl = rouge_f1(predictions, gold_explanations, n='L')
    def eval_sgc(exp, claim): return all(entails(e, claim) for e in exp)
    def eval_wgc(exp, claim): return all(not entails(e, neg(claim)) for e in exp)
    def eval_lc(exp):
        sents = exp.split('. ')
        return all(not entails(a, neg(b)) for a in sents for b in sents)
    return macro_f1, r1, r2, rl, eval_sgc, eval_wgc, eval_lc
```

## Common pitfalls

- ROUGE scores are a poor proxy for explanation quality because generated explanations are heterogeneous and not directly comparable to gold standards.
- Computational NLI evaluation reliably approximates weak global and local coherence but fails to approximate strong global coherence, as entailment is a weak proxy for the required property.
- Human evaluation was conducted on a small sample (25 test examples with 5 annotators), limiting statistical reliability.

## Evidence (verbatim from paper)

> We evaluated veracity prediction using macroF1, precision, recall and accuracy metrics as shown in Table 4.

## Citation

```bibtex
@misc{kotonya2020explainable,
  title={Explainable Automated Fact-Checking for Public Health Claims},
  author={Kotonya et al. (2020)},
  year={2020},
  note={arXiv:2010.09926}
}
```

- arXiv: 2010.09926