Quantemp Eval

---
name: quantemp-eval
description: Evaluates a model's ability to fact-check real-world numerical claims containing statistical and temporal expressions. It probes evidence retrieval, claim decomposition, and natural language inference to predict veracity (True, False, or Conflicting). Use when the user wants to benchmark on NumTemp, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17169
  bibtex_key: venktesh2024quantemp
  confidence: high
---

# quantemp-eval

> QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims — Venktesh V et al. (arXiv:2403.17169, 2024)

## What this evaluates

Evaluates a model's ability to fact-check real-world numerical claims containing statistical and temporal expressions. It probes evidence retrieval, claim decomposition, and natural language inference to predict veracity (True, False, or Conflicting).

## Datasets

- **NumTemp** — total 15514; splits: train (-1), val (-1), test (-1)

## Metrics

- `Macro-F1` **(primary)** — range: percent
  - Unweighted mean of the F1 scores for each of the three veracity classes (True, False, Conflicting). Calculated as the average of per-class precision-recall F1 scores.
- `Weighted-F1` — range: percent
  - Weighted mean of the F1 scores for each veracity class, where weights correspond to the number of true instances for each class.

## Input / output format

**Input**: Claim text (optionally decomposed into sub-questions or program steps) concatenated with retrieved evidence snippets (top 3) using separators.

**Output**: Veracity label: True, False, or Conflicting.

## Scoring recipe

```python
def compute_macro_f1(preds, gold):
    classes = ['True', 'False', 'Conflicting']
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- Evidence leakage: Using ground-truth justification paragraphs directly for retrieval instead of simulating a real-world retrieval pipeline.
- Training on mixed or non-numerical claims significantly degrades performance on numerical verification tasks.
- Ignoring claim decomposition leads to poor retrieval of quantitative evidence, hurting F1 scores.

## Evidence (verbatim from paper)

> Table 2: Results of different models on NumTemp (categorical and full) with Roberta-Large-MNLI as the NLI model. M-F1 : Macro-F1, W-F1 : Weighted-F1 and C-F1 refers to F1 score for Conflicting class.

## Citation

```bibtex
@misc{venktesh2024quantemp,
  title={QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims},
  author={Venktesh V et al.},
  year={2024},
  note={arXiv:2403.17169}
}
```

- arXiv: 2403.17169