Piqa Eval

---
name: piqa-eval
description: Evaluates a model's ability to reason about physical commonsense and intuitive physics by selecting the correct solution for a given goal from two options. It probes understanding of object affordances, material properties, and non-prototypical uses of everyday items. Use when the user wants to benchmark on PIQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.11641
  bibtex_key: bisk2019piqa
  confidence: high
---

# piqa-eval

> PIQA: Reasoning about Physical Commonsense in Natural Language — Bisk et al. (2019) (arXiv:1911.11641, 2019)

## What this evaluates

Evaluates a model's ability to reason about physical commonsense and intuitive physics by selecting the correct solution for a given goal from two options. It probes understanding of object affordances, material properties, and non-prototypical uses of everyday items.

## Datasets

- **PIQA** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options out of the total number of instances. Computed as a binary classification accuracy over two candidate solutions.

## Input / output format

**Input**: A goal description followed by two candidate solutions (Option A and Option B), processed with a [CLS] token for each choice.

**Output**: A binary classification prediction indicating whether Option A or Option B is the correct solution.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Some model 'mistakes' are actually correct but require external web search to verify, making strict accuracy evaluation potentially misleading.
- Finetuning can be highly unstable with certain hyperparameters, sometimes dropping validation performance to chance levels.
- Human performance was only evaluated on the development set, not the test set, so direct human-to-model comparison on test data is not available.

## Evidence (verbatim from paper)

> Table 1: Results of state-of-the-art natural language understanding models on PIQA, compared with human performance. The results show a significant gap between model and human performance, of roughly 20 absolute points. Accuracy (%) Validation Test

## Citation

```bibtex
@misc{bisk2019piqa,
  title={PIQA: Reasoning about Physical Commonsense in Natural Language},
  author={Bisk et al. (2019)},
  year={2019},
  note={arXiv:1911.11641}
}
```

- arXiv: 1911.11641