Zero Shot Commonsense Eval

---
name: zero-shot-commonsense-eval
description: Evaluates zero-shot commonsense reasoning capabilities of language models using multiple-choice questions. It specifically probes how prompt engineering and probability calibration strategies affect accuracy across different model sizes and architectures. Use when the user wants to benchmark on CommonsenseQA, COPA, OpenBookQA, PIQA, Social IQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.06962
  bibtex_key: ma2023promptengineering
  confidence: high
---

# zero-shot-commonsense-eval

> Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning — Ma (2023) (arXiv:2304.06962, 2023)

## What this evaluates

Evaluates zero-shot commonsense reasoning capabilities of language models using multiple-choice questions. It specifically probes how prompt engineering and probability calibration strategies affect accuracy across different model sizes and architectures.

## Datasets

- **CommonsenseQA** — total ?; splits: dev (-1)
- **COPA** — total ?; splits: dev (-1)
- **OpenBookQA** — total ?; splits: dev (-1)
- **PIQA** — total ?; splits: dev (-1)
- **Social IQA** — total ?; splits: dev (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Number of correctly predicted options divided by total number of instances, expressed as a percentage.

## Input / output format

**Input**: Multiple-choice question with a set of candidate options.

**Output**: The model selects the option with the highest conditional probability score.

## Scoring recipe

```python
correct = 0
for instance in dataset:
    probs = [model.logit(option) for option in instance.options]
    pred = instance.options[argmax(probs)]
    if pred == instance.gold:
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Combining prompt engineering and calibration often degrades performance rather than improving it due to negative interactions.
- Calibration benefits vary significantly across model architectures (e.g., helps GPT-2 but inconsistently helps T5).
- Prompt engineering only yields significant gains on larger, instruction-tuned models like Flan-T5-XL.

## Evidence (verbatim from paper)

> We evaluate prompt engineering and calibration on five multiple choice commonsense benchmarks: (1) CommonsenseQA (CSQA) (Talmor et al., 2019); (2) COPA (Gordon et al., 2012); (3) OpenBookQA (OBQA) (Mihaylov et al., 2018); (4)PIQA (Bisk et al., 2019); (5)Social IQA (SIQA) (Sap et al., 2019); We present their statistics in Appendix B. For all benchmarks, we only use their development sets. ... The evaluation metric is accuracy.

## Citation

```bibtex
@misc{ma2023promptengineering,
  title={Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning},
  author={Ma (2023)},
  year={2023},
  note={arXiv:2304.06962}
}
```

- arXiv: 2304.06962