Fastfact Eval

---
name: fastfact-eval
description: Evaluates the long-form factuality of LLM-generated responses by extracting claims, verifying them against evidence, and comparing the system's factuality scores against human annotations. Use when the user wants to benchmark on FaStfact-Bench, or asks about evaluating this task. Reports F₁@K′.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.12839
  bibtex_key: wan2025fastfact
  confidence: high
---

# fastfact-eval

> FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs — Wan et al. (2025) (arXiv:2510.12839, 2025)

## What this evaluates

Evaluates the long-form factuality of LLM-generated responses by extracting claims, verifying them against evidence, and comparing the system's factuality scores against human annotations.

## Datasets

- **FaStfact-Bench** — total 400; splits: test (400); repo https://github.com/Yingjia-Wan/FastFact

## Metrics

- `F₁@K′` **(primary)** — range: [0, 1]
  - F1 score computed at a threshold K′, measuring the overlap between predicted and ground-truth claim counts/scores.

## Input / output format

**Input**: Long-form QA pairs (question + generated response) with ground-truth human annotations for claim counts and factuality.

**Output**: Predicted factuality score (F₁@K′), extracted claim count, and total token cost per sample.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    pred_claims = extract_claims(predictions['response'])
    gt_claims = gold['claim_count']
    f1_at_k_prime = f1_score(pred_claims, gt_claims, threshold=K_prime)
    delta_k = abs(pred_claims - gt_claims)
    delta_f1 = abs(f1_at_k_prime - gold['f1_score'])
    token_cost = count_tokens(predictions['response'])
    return f1_at_k_prime, delta_k, delta_f1, token_cost
```

## Common pitfalls

- Using sentence-level chunking (stride=1) causes over-decomposition, inflating claim counts and token costs without improving accuracy.
- Token cost is reported as the primary efficiency proxy, which may not reflect actual inference latency or wall-clock time.
- Alignment metrics ($|\Delta K|$ and $|\Delta F_{1}@K^{\prime}|$) measure absolute deviation, potentially masking systematic bias in claim extraction.

## Evidence (verbatim from paper)

> By utilizing the ground-truth annotations from FaStfact-Bench, we run FaStfact and several baselines in parallel using the same underlying LLM as the extractor and verifier to compare their (1) evaluation reliability, proxied by the alignment with human judgment average $|\Delta K|$ and $|\Delta$$F_{1}@K^{\prime}$ $|$, and (2) practical efficiency, proxied by total token cost per sample.

## Citation

```bibtex
@misc{wan2025fastfact,
  title={FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs},
  author={Wan et al. (2025)},
  year={2025},
  note={arXiv:2510.12839}
}
```

- arXiv: 2510.12839