Global Piqa Eval

---
name: global-piqa-eval
description: This benchmark probes physical commonsense reasoning by testing whether models can distinguish correct from incorrect solutions to everyday physical tasks. It specifically evaluates cultural and linguistic grounding by using items constructed natively in 116 language varieties, avoiding translation artifacts that often skew multilingual evaluations. Use when the user wants to benchmark on Global PIQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.24081
  bibtex_key: chang2025globalpiqa
  confidence: high
---

# global-piqa-eval

> Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures — Chang et al. (2025) (arXiv:2510.24081, 2025)

## What this evaluates

This benchmark probes physical commonsense reasoning by testing whether models can distinguish correct from incorrect solutions to everyday physical tasks. It specifically evaluates cultural and linguistic grounding by using items constructed natively in 116 language varieties, avoiding translation artifacts that often skew multilingual evaluations.

## Datasets

- **Global PIQA** — total ?; splits: non-parallel (-1); repo https://github.com/mrlbenchmarks/global-piqa

## Metrics

- `accuracy` **(primary)** — range: percent
  - Mark correct if the model's predicted solution matches the ground truth via exact string matching (prompted format) or if the correct candidate has a higher normalized log-probability than the incorrect candidate (completion format). Chance accuracy is 50%.

## Input / output format

**Input**: A prompt containing a physical commonsense scenario or question in a specific language, followed by two candidate solutions (one correct, one incorrect).

**Output**: For prompted evaluation: a text response sampled up to 2048 tokens. For completion evaluation: log-probabilities for each candidate solution.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_sol in zip(predictions, gold):
        if pred.strip() == gold_sol.strip():
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Using the prompted format for base/pretrained models obscures true capabilities because instruction-following imposes auxiliary task demands that lower scores.
- Reporting only aggregate accuracy masks severe performance disparities across regions, with some low-resource languages dropping up to 37% below high-resource baselines.

## Evidence (verbatim from paper)

> For models that are tuned to follow instructions (e.g. the vast majority of proprietary models, and instruction-tuned and RL-tuned open models), we prompt the LLM with the prompt template in Figure[2]. We sample up to 2048 tokens, and score the responses using string matching. ... For all results, we report accuracy, where chance accuracy is 50%.

## Citation

```bibtex
@misc{chang2025globalpiqa,
  title={Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures},
  author={Chang et al. (2025)},
  year={2025},
  note={arXiv:2510.24081}
}
```

- arXiv: 2510.24081