Faviq Eval

---
name: faviq-eval
description: Evaluates a model's ability to verify factual claims against retrieved evidence, specifically focusing on claims derived from ambiguous information-seeking questions. It also measures the effectiveness of transfer learning from crowdsourced fact-checking data to professional fact-checking benchmarks. Use when the user wants to benchmark on FaVIQ, Snopes, SciFACT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.02153
  bibtex_key: park2021faviq
  confidence: high
---

# faviq-eval

> FaVIQ: FAct Verification from Information-seeking Questions — Park et al. (2021) (arXiv:2107.02153, 2021)

## What this evaluates

Evaluates a model's ability to verify factual claims against retrieved evidence, specifically focusing on claims derived from ambiguous information-seeking questions. It also measures the effectiveness of transfer learning from crowdsourced fact-checking data to professional fact-checking benchmarks.

## Datasets

- **FaVIQ** — total 188000; splits: train (-1), dev (-1), test (-1); repo https://github.com/faviq/faviq
- **Snopes** — total 6422; splits: test (-1), dev (-1)
- **SciFACT** — total 1109; splits: test (-1), dev (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly predicted verdicts (support or refute) out of the total number of claims. The task is framed as a 2-way classification where 'not enough info' (NEI) is merged into 'refute'.

## Input / output format

**Input**: A claim concatenated with up to k retrieved passages (e.g., k=3 Wikipedia passages or k=10 scientific abstracts).

**Output**: Generate either 'support' or 'refute'.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Models trained on FEVER perform poorly on FaVIQ in zero-shot (near random), highlighting a severe domain shift and lack of generalization to ambiguous claims.
- Retrieval errors account for 38% of failures, often because claims have low lexical overlap with the correct evidence passages.
- The original 3-way classification (Support/Refute/NEI) is converted to 2-way by merging NEI into Refute, which changes the evaluation landscape compared to prior work.

## Evidence (verbatim from paper)

> The overall accuracy of the baselines is low, despite their high performance on FEVER. The zero-shot performance is barely better than random guessing, indicating that the model trained on FEVER is not able to generalize to our more challenging data.

## Citation

```bibtex
@misc{park2021faviq,
  title={FaVIQ: FAct Verification from Information-seeking Questions},
  author={Park et al. (2021)},
  year={2021},
  note={arXiv:2107.02153}
}
```

- arXiv: 2107.02153