Geoparsing Eval

---
name: geoparsing-eval
description: This benchmark evaluates a model's ability to perceive and parse geometric diagrams into a structured formal language. It probes fine-grained visual primitive detection (points, lines, circles, planes) and spatial/semantic relations, testing both syntactic correctness and holistic geometric consistency. Use when the user wants to benchmark on GDP-29K, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11600
  bibtex_key: wang2026geoparsing
  confidence: high
---

# geoparsing-eval

> Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language — Wang et al. (2026) (arXiv:2604.11600, 2026)

## What this evaluates

This benchmark evaluates a model's ability to perceive and parse geometric diagrams into a structured formal language. It probes fine-grained visual primitive detection (points, lines, circles, planes) and spatial/semantic relations, testing both syntactic correctness and holistic geometric consistency.

## Datasets

- **GDP-29K** — total 29000; splits: train (26000), test (3000)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Computed per primitive category by comparing predicted and ground-truth sets of geometric primitives and relations. F1 = 2 * (Precision * Recall) / (Precision + Recall).
- `Sample Accuracy (SA)` — range: percent
  - Binary metric indicating whether a single diagram is parsed perfectly as a whole. SA = 1 if the predicted formal description exactly matches the ground-truth description, else 0.
- `Perfect Parsing Rate (PPR)` — range: percent
  - The fraction of samples in a benchmark subset that achieve a Sample Accuracy of 1, measuring holistic diagram-level correctness.

## Input / output format

**Input**: RGB images of plane or solid geometry diagrams.

**Output**: A unified formal language string representing the diagram's geometric primitives and spatial/semantic relations.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    p_list, r_list, f1_list, sa_list = [], [], [], []
    for pred_set, gold_set in zip(predictions, golds):
        tp = len(pred_set & gold_set)
        p = tp / len(pred_set) if pred_set else 0
        r = tp / len(gold_set) if gold_set else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        p_list.append(p); r_list.append(r); f1_list.append(f1)
        sa_list.append(1.0 if pred_set == gold_set else 0.0)
    return {
        'P': sum(p_list) / len(p_list),
        'R': sum(r_list) / len(r_list),
        'F1': sum(f1_list) / len(f1_list),
        'SA': sum(sa_list) / len(sa_list),
        'PPR': sum(sa_list) / len(sa_list)
    }
```

## Common pitfalls

- High category-level F1 scores can mask holistic failures; a single primitive error invalidates the entire diagram's formal description (multiplier effect).
- Baseline models often detect basic primitives well but fail on higher-order semantic relations (e.g., parallelism, angles), requiring specialized training beyond general visual pre-training.
- Downstream reasoning gains may be underestimated on solid geometry benchmarks due to textual explicitness in problem statements, which reduces the marginal utility of parsed diagrams.

## Evidence (verbatim from paper)

> While category-level F1 measures fine-grained parsing quality, it does not necessarily indicate that a diagram is parsed perfectly as a whole. To better evaluate holistic correctness, we additionally report Sample Accuracy (SA) for each category and Perfect Parsing Rate (PPR) for the full diagram.

## Citation

```bibtex
@misc{wang2026geoparsing,
  title={Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.11600}
}
```

- arXiv: 2604.11600