User Claim Distribution Eval

---
name: user-claim-distribution-eval
description: This evaluation probes the semantic, epistemic, and verifiability characteristics of real-world user-submitted fact-checking requests. It measures how public demand for verification aligns with or diverges from synthetic benchmark distributions, highlighting gaps in current misinformation evaluation corpora. Use when the user wants to benchmark on User Fact-Checking Claims Dataset, or asks about evaluating this task. Reports Veracity Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10935
  bibtex_key: ghafouri2026what
  confidence: high
---

# user-claim-distribution-eval

> What do people want to fact-check? — Ghafouri et al. (2026) (arXiv:2602.10935, 2026)

## What this evaluates

This evaluation probes the semantic, epistemic, and verifiability characteristics of real-world user-submitted fact-checking requests. It measures how public demand for verification aligns with or diverges from synthetic benchmark distributions, highlighting gaps in current misinformation evaluation corpora.

## Datasets

- **User Fact-Checking Claims Dataset** — total ?; splits: full (-1)

## Metrics

- `Shannon Entropy` — range: other
  - H_i^(d) = -sum_{k=1}^{K_d} p_{ik}^{(d)} log p_{ik}^{(d)}, where p_{ik}^{(d)} is the proportion of user i's claims in category k of dimension d. Measures user-level diversity across classification dimensions.
- `Veracity Score` **(primary)** — range: [0, 1]
  - A continuous measure from 0 (false) to 1 (true) reflecting how definitively a claim can be resolved. Computed per claim and averaged across dimensions.
- `Classification Proportion` — range: percent
  - Percentage of claims falling into each of the five semantic dimensions: Domain, Epistemic Type, Verifiability Level, Target Entity, and Temporal Orientation.

## Input / output format

**Input**: Raw text of user-submitted fact-checking claims.

**Output**: Five categorical labels (Domain, Epistemic Type, Verifiability, Target, Temporal), a continuous veracity score (0-1), and per-user entropy scores across dimensions.

## Scoring recipe

```python
def compute_metrics(claims, classifiers):
    # 1. Classify each claim across 5 dimensions
    labels = {dim: classifier(cl) for dim, classifier in classifiers.items()}
    # 2. Compute continuous veracity score (0-1) per claim
    veracity = veracity_model(cl)
    # 3. Compute user-level Shannon entropy per dimension
    user_dist = count_proportions(user_claims, dimension)
    entropy = -sum(p * log(p) for p in user_dist.values())
    return labels, veracity, entropy
```

## Common pitfalls

- Assuming veracity is binary; the paper uses a continuous 0-1 scale reflecting resolution definitiveness.
- Overlooking that ~25% of claims are normative or unverifiable, which standard benchmarks like FEVER typically exclude or misrepresent.
- Treating the dataset as a model training benchmark rather than a demand-side distribution analysis.

## Evidence (verbatim from paper)

> We then compared the average veracity score in each dataset, a continuous measure from 0 (false) to 1 (true) reflecting how definitively a claim can be resolved, and find that user-submitted claims cluster much closer to the ambiguous middle (mean = 0.58, SD = 0.38) than FEVER’s sharply polarized claims (mean = 0.38, SD = 0.47), showing that real users most often seek verification for grey-area statements rather than the clearly true or false claims that dominate benchmark datasets.

## Citation

```bibtex
@misc{ghafouri2026what,
  title={What do people want to fact-check?},
  author={Ghafouri et al. (2026)},
  year={2026},
  note={arXiv:2602.10935}
}
```

- arXiv: 2602.10935