Forgotten Polygons Eval

---
name: forgotten-polygons-eval
description: This benchmark probes the ability of multimodal large language models to recognize regular and irregular geometric shapes from images and accurately count their sides. It further evaluates multi-step visual-mathematical reasoning by requiring models to identify multiple shapes, map them to side counts, and compute their sum. Use when the user wants to benchmark on Forgotten Polygons, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.15969
  bibtex_key: [REDACTED]
  confidence: high
---

# forgotten-polygons-eval

> Forgotten Polygons: Multimodal Large Language Models are Shape-Blind — Rudman et al. (2025) (arXiv:2502.15969, 2025)

## What this evaluates

This benchmark probes the ability of multimodal large language models to recognize regular and irregular geometric shapes from images and accurately count their sides. It further evaluates multi-step visual-mathematical reasoning by requiring models to identify multiple shapes, map them to side counts, and compute their sum.

## Datasets

- **Forgotten Polygons** — total 2000; splits: test (-1); repo https://github.com/rsinghlab/Shape-Blind

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted shape names or side counts out of the total number of instances. Calculated as (correct predictions / total instances) * 100.

## Input / output format

**Input**: A 400×400 pixel image containing one or two geometric shapes (regular polygons, irregular polygons, or abstract shapes) with variations in color, rotation, size, and background. Accompanied by text prompts such as 'What shape is in the image?' or 'How many sides does the shape in the image have?'.

**Output**: Text response containing the shape name (e.g., 'triangle', 'pentagon') or an integer representing the number of sides.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if isinstance(gold, int):
            try:
                if int(pred) == gold: correct += 1
            except ValueError: pass
        else:
            if pred.strip().lower() == gold.strip().lower(): correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Confusing shape identification accuracy with side-counting accuracy, as models often recognize familiar shapes correctly but fail to count their sides (e.g., GPT-4o identifies heptagons at 92% but counts sides at 58%).
- Attributing low multi-step reasoning scores to arithmetic failures; the protocol shows Step 3 errors are primarily caused by incorrect shape identification or side mapping in earlier steps, not summation mistakes.
- Assuming language model backbones lack geometric knowledge; text-only prompts reveal near-perfect accuracy, indicating failures stem from vision encoders rather than textual pretraining gaps.

## Evidence (verbatim from paper)

> Table[1] shows that MLLMs excel at identifying common shapes like triangles and squares, with many models achieving perfect accuracy. However, performance drops sharply for less familiar polygons such as pentagons, heptagons, and octagons.

## Citation

```bibtex
@misc{rudman2025forgottenpolygons,
  title={Forgotten Polygons: Multimodal Large Language Models are Shape-Blind},
  author={Rudman et al. (2025)},
  year={2025},
  note={arXiv:2502.15969}
}
```

- arXiv: 2502.15969