Simpleqa Verified Eval

---
name: simpleqa-verified-eval
description: This benchmark evaluates an LLM's parametric factuality and internal knowledge recall on short-form questions. It measures whether models can correctly answer factual queries without relying on external search tools or retrieval augmentations. Use when the user wants to benchmark on SimpleQA Verified, or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.07968
  bibtex_key: haas2025simpleqa
  confidence: high
---

# simpleqa-verified-eval

> SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge — Haas et al. (2025) (arXiv:2509.07968, 2025)

## What this evaluates

This benchmark evaluates an LLM's parametric factuality and internal knowledge recall on short-form questions. It measures whether models can correctly answer factual queries without relying on external search tools or retrieval augmentations.

## Datasets

- **SimpleQA Verified** — total 1000; splits: test (1000)

## Metrics

- `F1-Score` **(primary)** — range: percent
  - Harmonic mean of Accuracy and Attempted rate, or computed via precision/recall from autorater judgments. Balances correctness with the model's willingness to attempt an answer.
- `Accuracy` — range: percent
  - Percentage of questions answered correctly out of all questions in the benchmark.
- `Acc.|Attempted` — range: percent
  - Percentage of correctly answered questions out of only those the model attempted to answer (excluding refusals or heavy hedging).

## Input / output format

**Input**: A single short-form factual question.

**Output**: A short textual answer, refusal, or hedge. Models are evaluated without external tools.

## Scoring recipe

```python
N = len(dataset)
for each question in dataset:
    gold = get_gold_answer(question)
    pred = model.generate(question, tools=False)
    is_correct = autorater_judge(pred, gold) == 'correct'
    is_attempted = not is_hedge_or_refusal(pred)
    accuracy = sum(is_correct) / N
    attempted_rate = sum(is_attempted) / N
    acc_attempted = sum(is_correct and is_attempted) / sum(is_attempted)
    f1 = 2 * (accuracy * attempted_rate) / (accuracy + attempted_rate) if (accuracy + attempted_rate) > 0 else 0
```

## Common pitfalls

- Enabling external tools or retrieval causes near-perfect scores, invalidating the parametric factuality measurement.
- Models that frequently hedge or refuse to answer (high Hedged rate) will show artificially low Accuracy but may retain high Acc.|Attempted, skewing overall performance interpretation.
- Results on SimpleQA Verified differ significantly from the original SimpleQA due to rigorous de-duplication and ground-truth correction, so cross-benchmark comparisons require caution.

## Evidence (verbatim from paper)

> In our results, we report the same metrics computed in Wei et al. ([2024a]) and use gpt-4.1-2025-04-14 as an autorater. Additionally, we measure the difference in scores between SimpleQA Verified and SimpleQA on all models. The results of GPT 4o, Claude Opus 4, Claude Sonnet 4 are statistically significantly worse on SimpleQA Verified compared to SimpleQA, whereas o4-mini’s score improves. On average, model performance on SimpleQA Verified is almost exactly the same as on SimpleQA Verified – our cleaning process described in [Section 2] removes erroneous and ambiguous questions which makes the benchmark easier. This is balanced by adversarially selecting a subset of challenging samples in [Section 2.6]. Gemini 2.5 Pro leads SimpleQA and SimpleQA Verified in both Accuracy and Accuracy Given Attempted (Acc.|Attempted) metrics, resulting in the highest F1-Score among frontier models.

## Citation

```bibtex
@misc{haas2025simpleqa,
  title={SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge},
  author={Haas et al. (2025)},
  year={2025},
  note={arXiv:2509.07968}
}
```

- arXiv: 2509.07968