Graph To Vision Eval

---
name: graph-to-vision-eval
description: This benchmark probes a vision-language model's ability to jointly interpret and reason across multiple related graph images. It specifically tests cross-modal integration, structural understanding, and instruction-following accuracy when processing homogeneous and heterogeneous graph groupings. Use when the user wants to benchmark on Graph-to-Vision Benchmark, or asks about evaluating this task. Reports instruction-following accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21435
  bibtex_key: li2025graphtovision
  confidence: medium
---

# graph-to-vision-eval

> Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models — Li et al. (2025) (arXiv:2503.21435, 2025)

## What this evaluates

This benchmark probes a vision-language model's ability to jointly interpret and reason across multiple related graph images. It specifically tests cross-modal integration, structural understanding, and instruction-following accuracy when processing homogeneous and heterogeneous graph groupings.

## Datasets

- **Graph-to-Vision Benchmark** — total 3459; splits: train (2769), val (390), test (300)

## Metrics

- `instruction-following accuracy` **(primary)** — range: percent
  - Percentage of model responses that exactly match or semantically align with the human-verified reference answer for each instruction.
- `reasoning consistency` — range: percent
  - Assessment of whether the model's multi-step reasoning across multiple graphs maintains logical coherence without contradictions.
- `graph parsing accuracy` — range: percent
  - Accuracy of correctly identifying and extracting graph elements (nodes, edges, structure) from the input images.

## Input / output format

**Input**: A set of interrelated graph images (flowcharts, knowledge graphs, mind maps, or route maps) paired with a natural language instruction requiring cross-graph reasoning.

**Output**: A natural language response answering the instruction, expected to demonstrate joint interpretation and integration of the provided graph structures.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # Calculate instruction-following accuracy
    exact_matches = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip())
    accuracy = exact_matches / len(predictions) * 100
    
    # Multi-dimensional framework for consistency and parsing
    # (Paper notes these are assessed via a multi-dimensional framework, likely LLM/human judging)
    consistency_scores = [evaluate_consistency(p) for p in predictions]
    parsing_scores = [evaluate_parsing(p) for p in predictions]
    
    return {
        'instruction_following_accuracy': accuracy,
        'reasoning_consistency': sum(consistency_scores) / len(consistency_scores),
        'graph_parsing_accuracy': sum(parsing_scores) / len(parsing_scores)
    }
```

## Common pitfalls

- Models must handle variable numbers of input images per prompt, which many standard VLMs are not optimized for.
- Heterogeneous graph groupings require cross-type reasoning, which can confuse models trained primarily on single-image or homogeneous multimodal data.
- GPT-4o generated instructions may contain implicit assumptions or biases that require careful human verification to ensure fair evaluation.

## Evidence (verbatim from paper)

> It evaluates state-of-the-art VLMs via a multi-dimensional framework assessing graph parsing, reasoning consistency, and instruction-following accuracy, and demonstrates consistent performance improvements through fine-tuning on open-source models, while highlighting limitations in scalability due to computational costs.

## Citation

```bibtex
@misc{li2025graphtovision,
  title={Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.21435}
}
```

- arXiv: 2503.21435