Geoqa Eval

---
name: geoqa-eval
description: Evaluates a model's ability to perform multimodal numerical reasoning on geometric problems by generating executable symbolic programs from text and diagram inputs. The model must fuse cross-modal information to predict step-by-step reasoning programs. These programs are then executed to select the correct multiple-choice answer from the given options. Use when the user wants to benchmark on GeoQA, or asks about evaluating this task. Reports answer accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.14517
  bibtex_key: chen2021geoqa
  confidence: high
---

# geoqa-eval

> GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning — Jiaqi Chen et al. (arXiv:2105.14517, 2021)

## What this evaluates

Evaluates a model's ability to perform multimodal numerical reasoning on geometric problems by generating executable symbolic programs from text and diagram inputs. The model must fuse cross-modal information to predict step-by-step reasoning programs. These programs are then executed to select the correct multiple-choice answer from the given options.

## Datasets

- **GeoQA** — total 4998; splits: test (-1); repo https://github.com/chen-judge/GeoQA

## Metrics

- `answer accuracy` **(primary)** — range: percent
  - Percentage of questions where the executed predicted program yields the correct multiple-choice option compared to the ground truth. Calculated as (correct predictions / total questions) * 100.

## Input / output format

**Input**: A geometric problem presented as a text description and a corresponding diagram image.

**Output**: A sequence of executable symbolic programs, which are executed to produce a final multiple-choice answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred_prog, gold_ans in zip(predictions, gold_answers):
        try:
            result = execute_program(pred_prog)
            if result == gold_ans:
                correct += 1
        except ExecutionError:
            pass
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Programs may execute to 'no result' (answer not in options or invalid syntax), which counts as incorrect and significantly lowers accuracy.
- Beam size heavily influences program generation and final accuracy; results must be reported with the specific beam size used (e.g., BS=10 vs BS=100).
- Text-only baselines perform poorly compared to multimodal ones; evaluating without diagram input misrepresents the task's requirements.

## Evidence (verbatim from paper)

> We conduct experiments on GeoQA dataset, and adopt answer accuracy as the evaluation metric. ... After the searched sequence program is executed, there will be three situations: right answer, wrong answer, and no result.

## Citation

```bibtex
@misc{chen2021geoqa,
  title={GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning},
  author={Jiaqi Chen et al.},
  year={2021},
  note={arXiv:2105.14517}
}
```

- arXiv: 2105.14517