Combigraph Vis Eval

---
name: combigraph-vis-eval
description: Evaluates multimodal discrete mathematical reasoning, specifically the ability to parse and solve combinatorial problems involving graphs, grids, and geometric diagrams. It also probes susceptibility to deliberately crafted distractors in multiple-choice formats versus genuine solution construction. Use when the user wants to benchmark on CombiGraph-Vis, or asks about evaluating this task. Reports avg@8.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.27094
  bibtex_key: mahdavi2025combigraphvis
  confidence: high
---

# combigraph-vis-eval

> CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning — Mahdavi et al. (2025) (arXiv:2510.27094, 2025)

## What this evaluates

Evaluates multimodal discrete mathematical reasoning, specifically the ability to parse and solve combinatorial problems involving graphs, grids, and geometric diagrams. It also probes susceptibility to deliberately crafted distractors in multiple-choice formats versus genuine solution construction.

## Datasets

- **CombiGraph-Vis** — total 1135; splits: test (1135); repo https://github.com/combigraphviz2025/combigraph-viz

## Metrics

- `avg@8` **(primary)** — range: [0, 1]
  - Mean correctness over 8 independent samples per problem. Calculated as the average of binary correctness scores across all generated samples for each instance, then averaged over the dataset.

## Input / output format

**Input**: A discrete mathematics problem statement, optionally accompanied by an image (e.g., graphs, grids, geometric diagrams), presented in either multiple-choice or yes/no format.

**Output**: A final answer, either a selected multiple-choice option or a yes/no response.

## Scoring recipe

```python
def compute_avg8(predictions_per_problem, gold_answers):
    correct_counts = []
    for preds, gold in zip(predictions_per_problem, gold_answers):
        correct = sum(1 for p in preds if p == gold)
        correct_counts.append(correct / 8)
    return sum(correct_counts) / len(correct_counts)
```

## Common pitfalls

- Models frequently select distractors in multiple-choice questions that are deliberately crafted to appear correct, inflating choice-matching rates without reflecting true reasoning.
- Performance drops significantly on image-tagged problems compared to text-only ones, so evaluating only text versions underestimates the benchmark's difficulty and modality gap.

## Evidence (verbatim from paper)

> For each model we compute: (i) Standalone avg@8 = mean correctness over 8 samples; and (ii) Among-Choices avg@8 = mean fraction of standalone problems whose final answer lies among the original options (not necessarily correct).

## Citation

```bibtex
@misc{mahdavi2025combigraphvis,
  title={CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning},
  author={Mahdavi et al. (2025)},
  year={2025},
  note={arXiv:2510.27094}
}
```

- arXiv: 2510.27094