Polymath Eval

---
name: polymath-eval
description: Evaluates multi-modal mathematical and cognitive reasoning capabilities on visual puzzles. It probes spatial interpretation, relational understanding, pattern recognition, and long-horizon logical reasoning using diagram-based multiple-choice questions. Use when the user wants to benchmark on POLYMATH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.14702
  bibtex_key: gupta2024polymath
  confidence: high
---

# polymath-eval

> Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark — Gupta et al. (2024) (arXiv:2410.14702, 2024)

## What this evaluates

Evaluates multi-modal mathematical and cognitive reasoning capabilities on visual puzzles. It probes spatial interpretation, relational understanding, pattern recognition, and long-horizon logical reasoning using diagram-based multiple-choice questions.

## Datasets

- **POLYMATH** — total 5000; splits: testmini (-1), test-img (-1); repo https://github.com/kevinscaria/PolyMATH

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact match accuracy calculated as the number of correctly predicted options divided by the total number of questions, reported as a percentage.

## Input / output format

**Input**: A visual puzzle (diagram/image) accompanied by a multiple-choice question. In ablation settings, the diagram is replaced with a detailed textual description.

**Output**: The model must generate a step-by-step solution, the final answer, and the corresponding multiple-choice option letter.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models significantly underperform on diagram-based questions compared to text-only descriptions, indicating poor visual comprehension rather than lack of reasoning ability.
- Accuracy varies drastically across prompting strategies (zero-shot vs. CoT vs. Step Back), so results are not directly comparable without specifying the prompt.
- Random chance baseline is not 25% but varies by category (8-16%) due to unequal option distributions or question types, making naive baselines misleading.

## Evidence (verbatim from paper)

> For multiple-choice questions, we use exact match for answer comparison. The model inference prompts are structured to elicit a step-by-step solution, the final answer, and the corresponding option. ... Across various prompting strategies (Table 3), Claude-3.5 Sonnet performed best with these advanced prompts, achieving up to 41.90% accuracy in Step Back Prompting, compared to 39.70% in zero shot.

## Citation

```bibtex
@misc{gupta2024polymath,
  title={Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark},
  author={Gupta et al. (2024)},
  year={2024},
  note={arXiv:2410.14702}
}
```

- arXiv: 2410.14702