Liar Eval

---
name: liar-eval
description: This evaluation probes a model's ability to perform binary fact-checking on short political claims by mapping multi-class truthfulness labels to positive/negative categories. It specifically tests how well the system handles compositional reasoning and uncertainty, requiring it to output definitive verdicts or abstain. Use when the user wants to benchmark on LIAR, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12184
  bibtex_key: gaddam2026trustagents
  confidence: high
---

# liar-eval

> TRUST Agents: A Collaborative Multi-Agent Framework for Fake News Detection, Explainable Verification, and Logic-Aware Claim Reasoning — Aishwarya Gaddam et al. (arXiv:2604.12184, 2026)

## What this evaluates

This evaluation probes a model's ability to perform binary fact-checking on short political claims by mapping multi-class truthfulness labels to positive/negative categories. It specifically tests how well the system handles compositional reasoning and uncertainty, requiring it to output definitive verdicts or abstain.

## Datasets

- **LIAR** — total 12836; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted binary labels out of the total number of instances.
- `macro-F1` — range: [0, 1]
  - Macro-averaged F1 score: the unweighted mean of the F1 scores for each class, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Short political claims/statement text.

**Output**: Binary truthfulness label ('true' or 'false') or 'uncertain'.

## Scoring recipe

```python
# Map gold labels to binary
pos_labels = {'true', 'mostly true', 'half true'}
neg_labels = {'false', 'pants on fire', 'barely true'}
gold_binary = ['pos' if g in pos_labels else 'neg' for g in gold_labels]

# Map model predictions to binary under two settings
pred_pess = ['neg' if p == 'uncertain' else ('pos' if p in pos_labels else 'neg') for p in preds]
pred_opt = ['pos' if p == 'uncertain' else ('pos' if p in pos_labels else 'neg') for p in preds]

# Compute metrics
acc_pess = sum(1 for g, p in zip(gold_binary, pred_pess) if g == p) / len(gold_binary)
f1_pess = macro_f1(gold_binary, pred_pess)
acc_opt = sum(1 for g, p in zip(gold_binary, pred_opt) if g == p) / len(gold_binary)
f1_opt = macro_f1(gold_binary, pred_opt)
```

## Common pitfalls

- The 'uncertain' output from the model must be arbitrarily mapped to either true or false for binary evaluation, which brackets performance but lacks a principled resolution.
- Class imbalance in the LIAR dataset means accuracy alone can be misleading; macro-F1 is required to properly evaluate minority class performance.

## Evidence (verbatim from paper)

> We report accuracy and macro-F1 as the main evaluation metrics. Accuracy gives a general measure of correctness, while macro-F1 is more informative under class imbalance and is less dominated by the majority class. Because TRUST Agents can output uncertain, we evaluate it under two mappings. In the pessimistic setting, uncertain is mapped to false. In the optimistic setting, uncertain is mapped to true.

## Citation

```bibtex
@misc{gaddam2026trustagents,
  title={TRUST Agents: A Collaborative Multi-Agent Framework for Fake News Detection, Explainable Verification, and Logic-Aware Claim Reasoning},
  author={Aishwarya Gaddam et al.},
  year={2026},
  note={arXiv:2604.12184}
}
```

- arXiv: 2604.12184