Chart To Code Eval

---
name: chart-to-code-eval
description: Evaluates a model's ability to translate chart images into executable plotting code, measuring both code executability and visual/textual fidelity of the generated charts. Use when the user wants to benchmark on ChartMimic, Plot2Code, ChartX, or asks about evaluating this task. Reports High-Level Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.06598
  bibtex_key: zhao2025chartcoder
  confidence: high
---

# chart-to-code-eval

> ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation — Zhao et al. (2025) (arXiv:2501.06598, 2025)

## What this evaluates

Evaluates a model's ability to translate chart images into executable plotting code, measuring both code executability and visual/textual fidelity of the generated charts.

## Datasets

- **ChartMimic** — total ?; splits: test (-1)
- **Plot2Code** — total ?; splits: test (-1)
- **ChartX** — total ?; splits: test (-1)

## Metrics

- `Exec.Rate` — range: percent
  - Percentage of generated code snippets that successfully execute without runtime errors.
- `Low-Level Score` — range: [0, 100]
  - Average similarity score between generated code and ground truth code across four aspects: text, layout, type, and color.
- `High-Level Score` **(primary)** — range: [0, 100]
  - Average similarity score between the chart image generated by executing the predicted code and the ground truth chart image, evaluated by GPT-4o across six aspects: chart types, layout, text content, data, style, and clarity.

## Input / output format

**Input**: Chart image

**Output**: Code snippet (e.g., Python/Matplotlib) to reproduce the chart

## Scoring recipe

```python
exec_rate = sum(1 for code in predictions if code.executes()) / len(predictions) * 100
low_level = 0
for pred, gold in zip(predictions, golds):
    low_level += compare_code_aspects(pred, gold, aspects=['text', 'layout', 'type', 'color'])
low_level /= len(predictions) * 4 * 100
high_level = 0
for pred, gold in zip(predictions, golds):
    generated_img = execute_code(pred)
    high_level += gpt4o_evaluate_similarity(generated_img, gold, aspects=['type', 'layout', 'text', 'data', 'style', 'clarity'])
high_level /= len(predictions) * 6 * 100
```

## Common pitfalls

- Original Plot2Code evaluation only considered charts corresponding to executable code, biasing results toward simple charts; the revised protocol includes all charts.
- High-level scores rely entirely on GPT-4o for visual similarity, which may introduce LLM-specific biases or inconsistencies compared to human judgment.
- Low-level scores measure code similarity rather than direct visual output matching, meaning different but functionally equivalent code may be penalized.

## Evidence (verbatim from paper)

> The high-level score utilizes GPT-4o to evaluate the detailed similarity between the ground truth and generated chart images in six aspects: chart types, layout, text content, data, style, and clarity. The low-level score is calculated based on a comparison between the ground truth and the generated code, focusing on the code similarities in four aspects: text, layout, type, and color.

## Citation

```bibtex
@misc{zhao2025chartcoder,
  title={ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2501.06598}
}
```

- arXiv: 2501.06598