Factlens Eval

---
name: factlens-eval
description: Evaluates the quality of fine-grained claim decomposition and the downstream impact of sub-claim quality on fact verification performance. It probes an LLM's ability to break complex claims into atomic, sufficient, and non-fabricated sub-claims, and measures how these sub-claim properties correlate with verification accuracy. Use when the user wants to benchmark on CoverBench, or asks about evaluating this task. Reports verification F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.05980
  bibtex_key: [REDACTED]
  confidence: high
---

# factlens-eval

> FactLens: Benchmarking Fine-Grained Fact Verification — Mitra et al. (2024) (arXiv:2411.05980, 2024)

## What this evaluates

Evaluates the quality of fine-grained claim decomposition and the downstream impact of sub-claim quality on fact verification performance. It probes an LLM's ability to break complex claims into atomic, sufficient, and non-fabricated sub-claims, and measures how these sub-claim properties correlate with verification accuracy.

## Datasets

- **CoverBench** — total 733; splits: eval (733); repo https://github.com/megagonlabs/factlens

## Metrics

- `atomicity` — range: [1, 3]
  - Categorical score (non-atomic-2, non-atomic-1, atomic) mapped to numerical values (1, 2, 3). Average across instances.
- `sufficiency` — range: other
  - Categorical score (low, medium, high) averaged across instances.
- `fabrication` — range: other
  - Categorical score (low, medium, high) averaged across instances.
- `coverage` — range: other
  - Categorical score (low, medium, high) averaged across instances.
- `redundancy` — range: other
  - Categorical score (low, medium, high) averaged across instances.
- `readability` — range: other
  - Categorical score (low, medium, high) averaged across instances.
- `verification F1` **(primary)** — range: [0, 1]
  - F1-score of the aggregated claim-level verification label against ground-truth claim labels.

## Input / output format

**Input**: Original complex claim and associated evidence/context from CoverBench.

**Output**: List of decomposed sub-claims, per-sub-claim verification labels (true/false), and an aggregated claim-level verification label.

## Scoring recipe

```python
sub_claims = decompose_claim(claim)
sub_labels = [verify(sub_claim, evidence) for sub_claim in sub_claims]
claim_label = 'false' if any(l == 'false' for l in sub_labels) else 'true'

# FactLens Evaluator metrics (mapped to 1-3 or low/med/high)
atomicity_score = map_to_1_2_3(atomicity_label)
sufficiency_score = map_to_1_2_3(sufficiency_label)
# ... similarly for fabrication, coverage, redundancy, readability

# Aggregation
avg_atomicity = mean(atomicity_score across 733 instances)
verification_f1 = f1_score(gold_claim_labels, claim_label)
```

## Common pitfalls

- Sub-claim verification labels are aggregated to the claim level using a strict 'any-false' rule, which may penalize models for minor sub-claim errors even if the overall claim is true.
- Metrics like atomicity and fabrication use categorical low/medium/high scores that are manually mapped to numerical values (1-3) for averaging, which can obscure distribution details.
- The evaluation isolates decomposition and verification from evidence retrieval, meaning performance does not reflect end-to-end retrieval-augmented pipelines.

## Evidence (verbatim from paper)

> For each instance, we map the ‘low’, ‘medium’, ‘high’ scores (‘non-atomic-2’, ‘non-atomic-1’, ‘atomic’ for atomicity) to numerical values (1, 2, 3 respectively), and report the average for each metric across all 733 instances in the CoverBench dataset. To compare the performance of fine-grained verification labels (for each sub-claim) with the holistic verification label, we aggregate the fine-grained labels as false if at least one of the fine-grained labels is also false; otherwise, the claim is considered true.

## Citation

```bibtex
@misc{mitra2024factlens,
  title={FactLens: Benchmarking Fine-Grained Fact Verification},
  author={Mitra et al. (2024)},
  year={2024},
  note={arXiv:2411.05980}
}
```

- arXiv: 2411.05980