Figedit Chart Editing Eval

---
name: figedit-chart-editing-eval
description: This benchmark evaluates the ability of vision-language and image-editing models to perform semantically correct, structure-aware modifications to scientific charts. It probes whether models can follow precise editing instructions while preserving data-encoding consistency, axis coherence, and legend integrity, rather than merely producing pixel-level visual similarity. Use when the user wants to benchmark on FigEdit, or asks about evaluating this task. Reports Instruction-following score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.00752
  bibtex_key: [REDACTED]
  confidence: high
---

# figedit-chart-editing-eval

> Charts Are Not Images: On the Challenges of Scientific Chart Editing — Li et al. (2025) (arXiv:2512.00752, 2025)

## What this evaluates

This benchmark evaluates the ability of vision-language and image-editing models to perform semantically correct, structure-aware modifications to scientific charts. It probes whether models can follow precise editing instructions while preserving data-encoding consistency, axis coherence, and legend integrity, rather than merely producing pixel-level visual similarity.

## Datasets

- **FigEdit** — total ?; splits: test (-1); repo https://github.com/adobe-research/figure-editing

## Metrics

- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring luminance, contrast, and structure correlation between the original and edited chart images.
- `PSNR` — range: dB
  - Peak Signal-to-Noise Ratio in decibels, calculated from the mean squared error between the original and edited images.
- `Instruction-following score` **(primary)** — range: percent
  - LLM-based evaluation score measuring whether the model successfully applied the requested semantic edit to the chart.
- `Content-preservation score` — range: percent
  - LLM-based evaluation score measuring whether unedited chart elements (axes, labels, data points) remain intact after the edit.
- `OCR accuracy` — range: percent
  - Character-level or word-level accuracy of text recognition in the edited chart compared to ground truth text.

## Input / output format

**Input**: A scientific chart image paired with a standardized text instruction specifying the desired edit (e.g., add/remove data points, change colors, modify labels).

**Output**: An edited chart image intended to reflect the requested semantic transformation.

## Scoring recipe

```python
def score_edit(original_img, edited_img, instruction, gt_text=None):
    ssim = compute_ssim(original_img, edited_img)
    psnr = compute_psnr(original_img, edited_img)
    ocr_acc = compute_ocr_accuracy(edited_img, gt_text)
    llm_prompt_if = f"Did the edit '{instruction}' correctly apply to the chart? Check data/axis integrity."
    instruction_score = llm_evaluate(llm_prompt_if, original_img, edited_img)
    llm_prompt_pres = "Are unedited elements preserved?"
    preservation_score = llm_evaluate(llm_prompt_pres, original_img, edited_img)
    return {'SSIM': ssim, 'PSNR': psnr, 'OCR': ocr_acc, 'Instruction': instruction_score, 'Preservation': preservation_score}
```

## Common pitfalls

- Relying exclusively on SSIM or PSNR, which remain high even when semantic instructions are ignored or data is corrupted.
- Assuming visual smoothness or high pixel similarity indicates successful instruction following, which masks structural or data-encoding failures.
- Failing to verify axis coherence, legend integrity, and exact data-point placement, which are critical for scientific chart validity.

## Evidence (verbatim from paper)

> Classic metrics such as SSIM and PSNR exaggerate the performance of pixel-oriented models like Imagen 4, while LLM-based scores and OCR accuracy reveal significant semantic errors.

## Citation

```bibtex
@misc{li2025chartsarenotimages,
  title={Charts Are Not Images: On the Challenges of Scientific Chart Editing},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.00752}
}
```

- arXiv: 2512.00752