Plot2code Eval

---
name: plot2code-eval
description: Evaluates multi-modal large language models' ability to visually interpret scientific plots and generate corresponding executable Python (matplotlib) code. It probes fine-grained visual reasoning, text extraction from dense plots, and precise code generation for data visualization. Use when the user wants to benchmark on Plot2Code, or asks about evaluating this task. Reports Pass Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.07990
  bibtex_key: wu2024plot2code
  confidence: high
---

# plot2code-eval

> Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots — Chengyue Wu et al. (2024) (arXiv:2405.07990, 2024)

## What this evaluates

Evaluates multi-modal large language models' ability to visually interpret scientific plots and generate corresponding executable Python (matplotlib) code. It probes fine-grained visual reasoning, text extraction from dense plots, and precise code generation for data visualization.

## Datasets

- **Plot2Code** — total 132; splits: test (-1)

## Metrics

- `Pass Rate` **(primary)** — range: percent
  - Percentage of generated code snippets that execute without errors and produce a valid plot.
- `Text-Match` — range: percent
  - Ratio measuring the textual similarity between the generated plot and the reference plot, typically computed via string or embedding overlap.
- `Rating` — range: [0, 10]
  - GPT-4V overall fidelity rating on a scale of 0 to 10, assessing how closely the generated image matches the reference plot visually.

## Input / output format

**Input**: A scientific plot image (matplotlib figure). In the 'Conditional Asking' setting, an additional text instruction describing the plot is appended.

**Output**: Executable Python code using matplotlib to reproduce the plot.

## Scoring recipe

```python
def score(predictions, golds):
    pass_count = 0
    text_matches = []
    ratings = []
    for pred, gold in zip(predictions, golds):
        if execute_code(pred):
            pass_count += 1
        text_matches.append(calculate_text_similarity(pred, gold))
        ratings.append(gpt4v_judge(pred, gold))
    return {
        'Pass Rate': pass_count / len(predictions) * 100,
        'Text-Match': sum(text_matches) / len(text_matches),
        'Rating': sum(ratings) / len(ratings)
    }
```

## Common pitfalls

- Models heavily rely on textual instructions rather than visual reasoning, causing pass rates to drop when instructions are added (Conditional Asking).
- Chain-of-Thought and Plan-and-Solve prompting strategies do not consistently improve performance over default prompts on this benchmark.
- High-resolution image inputs significantly boost performance, indicating standard low-resolution encoders miss fine plot details.

## Evidence (verbatim from paper)

> We showcase the quantitative results of (M)LLMs on our Plot2Code benchmark here. The code pass rate, text-match ratio, and GPT-4V overall rating for both direct asking and conditional asking scenarios are reported in Table 3.

## Citation

```bibtex
@misc{wu2024plot2code,
  title={Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots},
  author={Chengyue Wu et al. (2024)},
  year={2024},
  note={arXiv:2405.07990}
}
```

- arXiv: 2405.07990