Charte3 Eval

---
name: charte3-eval
description: Evaluates multimodal image-to-image editing models on chart editing tasks, measuring both low-level visual fidelity and high-level semantic correctness and consistency against editing instructions. Use when the user wants to benchmark on ChartE³, or asks about evaluating this task. Reports Correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.21694
  bibtex_key: li2026charte3
  confidence: high
---

# charte3-eval

> ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing — Li et al. (2026) (arXiv:2601.21694, 2026)

## What this evaluates

Evaluates multimodal image-to-image editing models on chart editing tasks, measuring both low-level visual fidelity and high-level semantic correctness and consistency against editing instructions.

## Datasets

- **ChartE³** — total 1200; splits: test (1200)

## Metrics

- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring luminance, contrast, and structure similarity between original and edited images.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio in decibels, measuring the ratio between maximum possible power of the signal and the power of corrupting noise.
- `CLIP` — range: [0, 1]
  - Cosine similarity between CLIP image embeddings of the original and edited chart images.
- `DINO` — range: [0, 1]
  - Cosine similarity between DINOv2 self-supervised feature embeddings of the original and edited images.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity; lower values indicate higher perceptual similarity.
- `Correctness` **(primary)** — range: other
  - GPT-based subjective score (1-5 scale) evaluating whether the intended editing instructions are correctly executed.
- `Consistency` — range: other
  - GPT-based subjective score (1-5 scale) evaluating the preservation of non-edited chart elements and structural coherence.

## Input / output format

**Input**: Original chart image paired with a multimodal editing instruction (text prompt).

**Output**: Edited chart image.

## Scoring recipe

```python
def score_model(predictions, gold_images, instructions):
    obj_scores = []
    for pred, gold in zip(predictions, gold_images):
        obj_scores.append({
            'SSIM': compute_ssim(pred, gold),
            'PSNR': compute_psnr(pred, gold),
            'CLIP': clip_similarity(pred, gold),
            'DINO': dino_similarity(pred, gold),
            'LPIPS': lpips_distance(pred, gold)
        })
    subj_scores = evaluate_gpt_subjective(predictions, instructions, gold_images)
    return obj_scores, subj_scores
```

## Common pitfalls

- GPT-based subjective scores (Correctness/Consistency) are used as a proxy for human evaluation, which may have alignment issues on edge cases.
- Objective metrics like SSIM/PSNR measure pixel-level fidelity but do not guarantee semantic correctness of chart data.
- LPIPS is lower-better, while others are higher-better, which can cause ranking confusion.

## Evidence (verbatim from paper)

> Table 5 summarizes the evaluation results of different chart image editing models on our benchmark, including five objective metrics and two subjective metrics. Models achieving higher scores on objective metrics (e.g., SSIM, CLIP, DINO, and LPIPS) also tend to obtain better Correctness and Consistency scores in GPT-aligned judgment.

## Citation

```bibtex
@misc{li2026charte3,
  title={ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.21694}
}
```

- arXiv: 2601.21694