Commonsenseqa Eval

---
name: commonsenseqa-eval
description: This benchmark evaluates a model's ability to answer multiple-choice questions that require real-world commonsense knowledge. It specifically probes whether models can distinguish a correct answer from semantically plausible but factually incorrect distractors based on spatial, causal, or physical reasoning. Use when the user wants to benchmark on CommonsenseQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.00937
  bibtex_key: talmor2018commonsenseqa
  confidence: high
---

# commonsenseqa-eval

> CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge — Talmor et al. (2018) (arXiv:1811.00937, 2018)

## What this evaluates

This benchmark evaluates a model's ability to answer multiple-choice questions that require real-world commonsense knowledge. It specifically probes whether models can distinguish a correct answer from semantically plausible but factually incorrect distractors based on spatial, causal, or physical reasoning.

## Datasets

- **CommonsenseQA** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of examples for which the model's prediction matches the gold label. Calculated as the number of correct predictions divided by the total number of examples.

## Input / output format

**Input**: A question prompt accompanied by multiple-choice options, including one correct answer and several semantically plausible distractors derived from ConceptNet.

**Output**: A single selected option from the provided multiple-choice list.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using the 'question concept split' instead of the primary 'random split' artificially inflates performance because models can memorize concept-answer mappings that appear in both train and test sets.
- Confusing the main benchmark with the 'SANITY' mode, which replaces hard distractors with random ConceptNet ones to establish a much easier baseline.
- Assuming retrieval-augmented approaches (e.g., using web snippets) significantly improve performance, as the paper demonstrates they yield only marginal gains.

## Evidence (verbatim from paper)

> We evaluate all models on the test set using accuracy (proportion of examples for which prediction is correct), and tune hyper-parameters for all trained models on the development set.

## Citation

```bibtex
@misc{talmor2018commonsenseqa,
  title={CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge},
  author={Talmor et al. (2018)},
  year={2018},
  note={arXiv:1811.00937}
}
```

- arXiv: 1811.00937