Vista Score Eval

---
name: vista-score-eval
description: Evaluates conversational factuality and hallucination detection in LLMs by decomposing dialogue turns into atomic claims, verifying them against reference texts and dialogue history, and categorizing unverifiable content. It measures how well models track factual consistency across sequential turns. Use when the user wants to benchmark on FaithDial, or asks about evaluating this task. Reports claim-level accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.27052
  bibtex_key: lewis2025vista
  confidence: high
---

# vista-score-eval

> VISTA Score: Verification In Sequential Turn-based Assessment — Ashley Lewis et al. (2025) (arXiv:2510.27052, 2025)

## What this evaluates

Evaluates conversational factuality and hallucination detection in LLMs by decomposing dialogue turns into atomic claims, verifying them against reference texts and dialogue history, and categorizing unverifiable content. It measures how well models track factual consistency across sequential turns.

## Datasets

- **FaithDial** — total ?; splits: test (-1)

## Metrics

- `claim-level accuracy` **(primary)** — range: percent
  - Proportion of atomic claims correctly classified as VERIFIED or UNVERIFIABLE (or fine-grained subcategories) against gold annotations.
- `macro-F1` — range: percent
  - Unweighted mean of the F1 score calculated per class across all verification categories.
- `turn-level accuracy` — range: percent
  - Proportion of dialogue turns correctly classified as verifiable or unverifiable.

## Input / output format

**Input**: Dialogue context (prior turns), current turn text, and retrieved reference document. The model must decompose the turn into atomic claims and assign a verification label.

**Output**: A classification label per atomic claim: VERIFIED, CONTRADICTED, LACKING_EVIDENCE, SUBJECTIVE, or ABSTAIN (merged to UNVERIFIABLE for baseline comparison).

## Scoring recipe

```python
def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    classes = set(gold)
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) else 0
        rec = tp / (tp + fn) if (tp + fn) else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) else 0)
    return {'claim_accuracy': acc, 'macro_f1': sum(f1s) / len(f1s)}
```

## Common pitfalls

- Prior benchmarks often misclassify unverifiable or out-of-scope content as hallucinations; VISTA explicitly separates these into UNVERIFIABLE subcategories.
- Ignoring dialogue history or background context during verification significantly drops accuracy, as models rely on sequential grounding to resolve ambiguities.

## Evidence (verbatim from paper)

> We report turn-level accuracy (verifiable vs. unverifiable), claim-level accuracy, and macro-F1. The majority baseline predicts the most frequent claim label (VERIFIED).

## Citation

```bibtex
@misc{lewis2025vista,
  title={VISTA Score: Verification In Sequential Turn-based Assessment},
  author={Ashley Lewis et al. (2025)},
  year={2025},
  note={arXiv:2510.27052}
}
```

- arXiv: 2510.27052