Trendfact Eval

---
name: trendfact-eval
description: Evaluates fact-checking systems across three sub-tasks: retrieving relevant evidence, verifying claim truthfulness, and generating natural language explanations. It specifically probes a model's Hotspot Perception Ability (HPA) by measuring how effectively it allocates reasoning effort and computational resources based on the real-world influence of trending claims. Use when the user wants to benchmark on TrendFact, or asks about evaluating this task. Reports F1-macro.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15135
  bibtex_key: zhang2024trendfact
  confidence: high
---

# trendfact-eval

> TrendFact: A Benchmark for Explainable Hotspot Perception in Fact-Checking with Natural Language Explanation — Zhang et al. (2024) (arXiv:2410.15135, 2024)

## What this evaluates

Evaluates fact-checking systems across three sub-tasks: retrieving relevant evidence, verifying claim truthfulness, and generating natural language explanations. It specifically probes a model's Hotspot Perception Ability (HPA) by measuring how effectively it allocates reasoning effort and computational resources based on the real-world influence of trending claims.

## Datasets

- **TrendFact** — total 7643; splits: test (-1); repo https://github.com/zxc123cc/TrendFact

## Metrics

- `R@k (k=1,2,3,5)` — range: [0, 1]
  - Recall at k: fraction of gold evidence items successfully retrieved within the top-k ranked results from the evidence library.
- `F1-macro` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score across verification classes. Calculated as the unweighted mean of per-class F1 scores (2 * precision * recall / (precision + recall)).
- `HCPI` — range: [0, 1]
  - Hotspot Claim Perception Index: evaluates how well a system allocates reasoning effort based on claim influence, factoring in verification accuracy and reasoning efficiency/time.
- `ECS` — range: [0, 1]
  - Explanation Consistency Score: measures the alignment between the generated explanation and the model's internal reasoning process.
- `BLEU-4` — range: [0, 1]
  - Standard 4-gram BLEU score comparing generated explanations to reference explanations.
- `ROUGE-1/2/L` — range: [0, 1]
  - ROUGE scores measuring overlap of unigrams, bigrams, and longest common subsequence between generated and reference explanations.
- `BERTScore` — range: [0, 1]
  - F1 score computed using contextual embeddings from BERT to measure semantic similarity between generated and reference explanations.

## Input / output format

**Input**: Claim text, associated metadata (e.g., influence/hotspot score), and a dynamic evidence library for retrieval.

**Output**: Top-k retrieved evidence snippets, a verification label (e.g., True/False/Unsupported), and a natural language explanation justifying the verdict.

## Scoring recipe

```python
def score(predictions, gold):
    # Retrieval
    r_at_k = [sum(1 for p in preds[:k] if p in gold_evidence) / len(gold_evidence) for k in [1,2,3,5]]
    # Verification
    tp = sum(1 for p, g in zip(preds, gold) if p == g == 'True')
    fp = sum(1 for p, g in zip(preds, gold) if p == 'True' and g != 'True')
    fn = sum(1 for p, g in zip(preds, gold) if p != 'True' and g == 'True')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_macro = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    # Explanation & HPA
    ecs = compute_consistency_score(predictions.explanations, predictions.reasoning_trace)
    hcp = compute_hotspot_perception(predictions.f1, predictions.reasoning_effort, gold.influence_scores)
    return {'R@k': r_at_k, 'F1-macro': f1_macro, 'ECS': ecs, 'HCPI': hcp}
```

## Common pitfalls

- HCPI is derived from verification results and reasoning efficiency, so errors in the verification step directly cascade and penalize the hotspot perception score.
- ECS evaluates consistency with internal reasoning rather than factual correctness, which may artificially inflate scores for models that generate verbose but logically aligned explanations.
- The evidence library is dynamically augmented and contains challenging, similar-looking items, making R@k scores incomparable to standard retrieval benchmarks.

## Evidence (verbatim from paper)

> For evidence retrieval task, we choose R@k, where k=1,2,3,5. For verification task, we choose F1-macro, Precision, Recall, and Accuracy. For explanation generation task, in addition to ECS, we also employ BLEU-4, ROUGE-(1, 2, L), and BERTScore. For assessing the HPA of fact-checking systems, we employ HCPI.

## Citation

```bibtex
@misc{zhang2024trendfact,
  title={TrendFact: A Benchmark for Explainable Hotspot Perception in Fact-Checking with Natural Language Explanation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.15135}
}
```

- arXiv: 2410.15135