Quan Temp Plus Eval

---
name: quan-temp-plus-eval
description: Evaluates open-domain numerical fact-checking by testing how well models can verify claims using decomposed queries and retrieved evidence. It probes the impact of claim decomposition quality on evidence retrieval and downstream NLI-based verification accuracy. Use when the user wants to benchmark on QuanTemp++, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22055
  bibtex_key: venktesh2025quantemp
  confidence: high
---

# quan-temp-plus-eval

> A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition — Venktesh et al. (2025) (arXiv:2510.22055, 2025)

## What this evaluates

Evaluates open-domain numerical fact-checking by testing how well models can verify claims using decomposed queries and retrieved evidence. It probes the impact of claim decomposition quality on evidence retrieval and downstream NLI-based verification accuracy.

## Datasets

- **QuanTemp++** — total 15514; splits: train (9935), val (3084), test (2495); repo https://github.com/VenkteshV/QuanTemp_Plus

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted veracity labels (True, False, Contradiction) out of the total number of claims.
- `macro-F1 (M-F1)` — range: [0, 1]
  - Unweighted mean of F1 scores for each class (True, False, Contradiction), treating all classes equally.
- `weighted-F1 (W-F1)` — range: [0, 1]
  - F1 score averaged over classes, weighted by the number of true instances for each class.

## Input / output format

**Input**: Claim text and top-k retrieved evidence snippets from the QuanTemp++ collection.

**Output**: Veracity label prediction: True, False, or Contradiction.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    # Per-class F1, macro-F1, and weighted-F1 are computed using standard classification metrics
    # e.g., sklearn.metrics.f1_score(gold_labels, predictions, average='macro')
    return {'accuracy': accuracy, 'macro_f1': macro_f1, 'weighted_f1': weighted_f1}
```

## Common pitfalls

- The evaluation uses a fixed top-k retrieval (k=3) for the main NLI verification step, so models must handle evidence ordering and potential noise in retrieved snippets rather than performing end-to-end retrieval from scratch.
- Per-class F1 distinguishes between True, False, and Contradiction, which differs from standard binary fact-checking benchmarks and requires careful handling of the NLI label space.
- Evidence collection is pre-computed via FCDecomp; the benchmark evaluates verification given these snippets, not the decomposition or retrieval process itself.

## Evidence (verbatim from paper)

> We employ Contriever as retrieval model afetr comparing several approaches like BM25, ANCE, Tas-b on validation set. We experiment with different values of k=1,3,5,7,10 for top-k evidence retrieval and observe k=3 to provide best NLI performance on validation set. For each of these settings, we evaluate the accuracy, per-class F1, macro-F1 (M-F1), and weighted-F1 (W-F1) scores.

## Citation

```bibtex
@misc{venktesh2025quantemp,
  title={A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition},
  author={Venktesh et al. (2025)},
  year={2025},
  note={arXiv:2510.22055}
}
```

- arXiv: 2510.22055