Trilemma Of Truth Eval

---
name: trilemma-of-truth-eval
description: Evaluates large language models' ability to distinguish factually true statements from factually false and unverifiable ('neither') statements. It probes both prompt-based output probabilities and internal hidden activations to measure veracity classification accuracy and uncertainty quantification. Use when the user wants to benchmark on Trilemma of Truth Datasets, or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23921
  bibtex_key: savcisens2025trilemma
  confidence: high
---

# trilemma-of-truth-eval

> The Trilemma of Truth in Large Language Models — Savcisens et al. (2025) (arXiv:2506.23921, 2025)

## What this evaluates

Evaluates large language models' ability to distinguish factually true statements from factually false and unverifiable ('neither') statements. It probes both prompt-based output probabilities and internal hidden activations to measure veracity classification accuracy and uncertainty quantification.

## Datasets

- **Trilemma of Truth Datasets** — total 22275; splits: test (-1); HF `carlomarxx/trilemma-of-truth`; repo https://github.com/carlomarxdk/trilemma-of-truth

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthew’s Correlation Coefficient for multiclass classification: MCC = (c*s - Σ(pk*tk)) / sqrt((s^2 - Σpk^2) * (s^2 - Σtk^2)), where c is correct predictions, s is total samples, pk and tk are predicted and true class counts.
- `W-MCC` — range: [-1, 1]
  - Weighted-MCC adjusts MCC by the acceptance rate to penalize excessive abstentions: W-MCC = MCC * (1 - (#abstained / #total_predictions)).

## Input / output format

**Input**: Instruction prompt concatenated with a statement (e.g., 'The city of X is located in Y.') for zero-shot evaluation; or token-level hidden activations from the LLM for probing methods.

**Output**: For zero-shot: token probabilities mapped to veracity labels {true, false, neither, abstain}. For probes: predicted veracity label {true, false, neither}.

## Scoring recipe

```python
import math
def compute_mcc(preds, gold):
    c = sum(p == g for p, g in zip(preds, gold))
    s = len(gold)
    pk = [sum(p == k for p in preds) for k in ['true','false','neither']]
    tk = [sum(g == k for g in gold) for k in ['true','false','neither']]
    num = c*s - sum(p*t for p,t in zip(pk,tk))
    den = math.sqrt((s**2 - sum(p**2 for p in pk)) * (s**2 - sum(t**2 for t in tk)))
    return num/den if den > 0 else 0.0

def compute_w_mcc(preds, gold, abstained_count):
    mcc = compute_mcc(preds, gold)
    total = len(preds)
    return mcc * (1 - abstained_count / total)
```

## Common pitfalls

- Treating 'neither' statements as false; they are synthetically generated unverifiable claims and must be classified as a separate class.
- Reporting standard MCC when probes abstain; the protocol requires Weighted-MCC (W-MCC) to penalize high abstention rates.
- Assuming linear probes capture veracity symmetrically; the paper notes asymmetry between truth/false signals and that nonlinear probes are necessary for RLHF-distilled models.

## Evidence (verbatim from paper)

> We use Matthew’s Correlation Coefficient (MCC) to summarize the statistical accuracy of probes. The multiclass MCC value is calculated using Eq.[14]. where c is the number of correct predictions, s is the total number of samples, K is the total number of classes, t_k is the number of k-class samples in the data set, and p_k is the number of times k-class was predicted. MCC = 1 indicates that a classifier predicted every instance correctly. MCC = 0 implies that the predictions are random. MCC = -1 indicates that the predictions are inversely correlated with the ground-truth labels. Zero-shot prompting, one-vs-all sAwMIL, and multiclass sAwMIL can abstain from making predictions. If a probe abstains too often, it suggests poor performance. For these cases, we use Weighted-MCC (W-MCC), where the acceptance rate serves as the weight (see Eq.[15]).

## Citation

```bibtex
@misc{savcisens2025trilemma,
  title={The Trilemma of Truth in Large Language Models},
  author={Savcisens et al. (2025)},
  year={2025},
  note={arXiv:2506.23921}
}
```

- arXiv: 2506.23921