Oecd Tabular Fact Checking Eval

---
name: oecd-tabular-fact-checking-eval
description: Evaluates an LLM's ability to verify factual claims against high-volume tabular data by measuring evidence retrieval (table and data level) and claim factuality classification. It also probes whether models rely on parametric knowledge versus active retrieval and reasoning over structured data. Use when the user wants to benchmark on OECD Tabular Fact-Checking Dataset, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.17232
  bibtex_key: devasier2026frame
  confidence: high
---

# oecd-tabular-fact-checking-eval

> Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data — Devasier et al. (2026) (arXiv:2601.17232, 2026)

## What this evaluates

Evaluates an LLM's ability to verify factual claims against high-volume tabular data by measuring evidence retrieval (table and data level) and claim factuality classification. It also probes whether models rely on parametric knowledge versus active retrieval and reasoning over structured data.

## Datasets

- **OECD Tabular Fact-Checking Dataset** — total 78503; splits: test (-1)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly predicted verdicts (True, False, or NEI) or correctly retrieved tables/data rows out of the total number of instances. Calculated as (Correct Predictions / Total Instances) * 100.
- `Precision, Recall, F1-score` — range: [0, 1]
  - Standard classification metrics for claim-knowledge alignment, measuring the model's ability to predict True/False based on internal knowledge versus the generated claims.
- `Human quality rate` — range: percent
  - Percentage of randomly sampled claims manually annotated as 'good' by native speakers.

## Input / output format

**Input**: A synthetic factual claim derived from a high-volume tabular dataset, accompanied by the corresponding table(s) or database context for evidence retrieval.

**Output**: A predicted verdict (True, False, or NEI), optionally accompanied by retrieved evidence (specific table or data rows) or a generated query.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_retrieval_accuracy(retrieved_ids, gold_ids):
    correct = sum(1 for r, g in zip(retrieved_ids, gold_ids) if r == g)
    return (correct / len(gold_ids)) * 100
```

## Common pitfalls

- LLM judges struggle with exact numeric values and may classify claims as False due to minor rounding differences unless explicitly prompted to allow tolerance.
- High rates of 'Not Enough Information' (NEI) predictions can skew overall accuracy metrics if not analyzed separately from True/False verdicts.
- Human annotation quality varies significantly across languages, with lower quality rates observed for non-English languages like Hindi.

## Evidence (verbatim from paper)

> Our baseline performed significantly better on both predicted and gold tables, with the thinking-enabled LLM consistently performing the best, with 36% and 20.4% accuracy on the gold and predicted tables, respectively.

## Citation

```bibtex
@misc{devasier2026frame,
  title={Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data},
  author={Devasier et al. (2026)},
  year={2026},
  note={arXiv:2601.17232}
}
```

- arXiv: 2601.17232