Boolq Eval

---
name: boolq-eval
description: This benchmark evaluates a model's ability to answer naturally occurring yes/no questions based on a provided passage. It probes complex inferential reasoning and non-factoid inference, requiring the model to go beyond simple keyword matching or shallow statistical features to determine entailment or contradiction between the question and the passage. Use when the user wants to benchmark on BoolQ, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.10044
  bibtex_key: clark2019boolq
  confidence: high
---

# boolq-eval

> BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions — Clark et al. (2019) (arXiv:1905.10044, 2019)

## What this evaluates

This benchmark evaluates a model's ability to answer naturally occurring yes/no questions based on a provided passage. It probes complex inferential reasoning and non-factoid inference, requiring the model to go beyond simple keyword matching or shallow statistical features to determine entailment or contradiction between the question and the passage.

## Datasets

- **BoolQ** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted yes/no answers out of the total number of instances in the evaluation split. Calculated as the number of correct predictions divided by the total number of predictions.

## Input / output format

**Input**: A natural language yes/no question paired with a multi-sentence passage (context) from which the answer must be inferred.

**Output**: A binary classification label: 'yes' or 'no'.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models can achieve ~62% accuracy by simply predicting 'yes' for every question due to class imbalance, so the majority-class baseline must be reported and compared against.
- Shallow features like word overlap or TF-IDF are insufficient and often perform worse than the majority baseline, indicating the task requires deep semantic reasoning rather than lexical matching.

## Evidence (verbatim from paper)

> In general, we found features such as word overlap or TF-IDF statistics were not sufficient to achieve better than the majority-class baseline accuracy (62.17% on the dev set).

## Citation

```bibtex
@misc{clark2019boolq,
  title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
  author={Clark et al. (2019)},
  year={2019},
  note={arXiv:1905.10044}
}
```

- arXiv: 1905.10044