Graphcheck Factcheck Eval

---
name: graphcheck-factcheck-eval
description: This evaluation probes a model's ability to perform multihop fact-checking over long-form documents and open-domain QA contexts. It measures how well the system identifies factual inconsistencies or supports claims by reasoning over complex, lengthy grounding texts across general and medical domains. Use when the user wants to benchmark on AggreFact-CNN, AggreFact-Xsum, Summeval, ExpertQA, COVID-Fact, SCIFact, PubHealth, or asks about evaluating this task. Reports balanced accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.16514
  bibtex_key: [REDACTED]
  confidence: high
---

# graphcheck-factcheck-eval

> GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking — Yingjian Chen et al. (arXiv:2502.16514, 2025)

## What this evaluates

This evaluation probes a model's ability to perform multihop fact-checking over long-form documents and open-domain QA contexts. It measures how well the system identifies factual inconsistencies or supports claims by reasoning over complex, lengthy grounding texts across general and medical domains.

## Datasets

- **AggreFact-CNN** — total ?; splits: test (-1)
- **AggreFact-Xsum** — total ?; splits: test (-1)
- **Summeval** — total ?; splits: test (-1)
- **ExpertQA** — total ?; splits: test (-1)
- **COVID-Fact** — total ?; splits: test (-1)
- **SCIFact** — total ?; splits: test (-1)
- **PubHealth** — total ?; splits: test (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: percent
  - The average of recall scores computed per class: (1/C) * Σ (TP_c / (TP_c + FN_c)), where C is the number of classes. This metric is used to mitigate performance skew caused by class imbalance in the benchmarks.

## Input / output format

**Input**: A claim and its corresponding grounding document (long-form text or QA context).

**Output**: A binary classification label indicating whether the claim is supported/refuted or true/false.

## Scoring recipe

```python
def balanced_accuracy(predictions, gold):
    classes = np.unique(gold)
    recalls = []
    for c in classes:
        mask = gold == c
        recalls.append(np.mean(predictions[mask] == c))
    return np.mean(recalls) * 100
```

## Common pitfalls

- Standard accuracy is misleading due to class imbalance in several benchmarks; balanced accuracy must be used instead.
- OverallAvg. is computed as the mean of per-dataset balanced accuracies, which can obscure performance variations on specific domains like medical fact-checking.

## Evidence (verbatim from paper)

> Considering the data imbalance in some benchmarks, models biased towards a particular class in predictions may not reflect their true performance. To address this, we follow previous approached*Liu et al. ([2023b]); Tang et al. ([2023])* and calculate balanced accuracy (BAcc).

## Citation

```bibtex
@misc{chen2025graphcheck,
  title={GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking},
  author={Yingjian Chen et al.},
  year={2025},
  note={arXiv:2502.16514}
}
```

- arXiv: 2502.16514