Instruction Editing Eval

---
name: instruction-editing-eval
description: Evaluates instruction-guided image editing models on their ability to modify an input image according to a text prompt. It measures semantic alignment with the prompt and visual fidelity to a ground-truth edit, plus human preference in pairwise comparisons. Use when the user wants to benchmark on MagicBrush (MagBr), ZONE, or asks about evaluating this task. Reports CLIP-T.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.18406
  bibtex_key: chen2025instructclip
  confidence: high
---

# instruction-editing-eval

> Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning — Chen et al. (2025) (arXiv:2503.18406, 2025)

## What this evaluates

Evaluates instruction-guided image editing models on their ability to modify an input image according to a text prompt. It measures semantic alignment with the prompt and visual fidelity to a ground-truth edit, plus human preference in pairwise comparisons.

## Datasets

- **MagicBrush (MagBr)** — total ?; splits: test (-1)
- **ZONE** — total ?; splits: test (-1)

## Metrics

- `CLIP-T` **(primary)** — range: [0, 1]
  - Cosine similarity between CLIP visual features of the generated image and CLIP text features of the target instruction: sim(CLIPvis(I^e'), CLIPtxt(p^e)).
- `CLIP-I` — range: [0, 1]
  - Cosine similarity between CLIP visual features of the generated image and CLIP visual features of the ground-truth edited image: sim(CLIPvis(I^e'), CLIPvis(I^e)).
- `DINO-I` — range: [0, 1]
  - Cosine similarity between DINOv2 visual features of the generated image and DINOv2 visual features of the ground-truth edited image: sim(DINOv2(I^e'), DINOv2(I^e)).

## Input / output format

**Input**: Original image (I^o) and edit instruction (p^e)

**Output**: Edited image (I^e)'

## Scoring recipe

```python
def compute_clip_t(gen_img, target_prompt):
    gen_feat = clip_vision_model(gen_img)
    txt_feat = clip_text_model(target_prompt)
    return cosine_similarity(gen_feat, txt_feat)

def compute_visual_sim(gen_img, gt_img, model):
    gen_feat = model(gen_img)
    gt_feat = model(gt_img)
    return cosine_similarity(gen_feat, gt_feat)
```

## Common pitfalls

- CLIP-T, CLIP-I, and DINO-I scores do not always correlate with human-perceived visual quality of edits.
- Ground-truth edited images (I^e) are unavailable for the ZONE benchmark, so CLIP-I and DINO-I cannot be computed for it.

## Evidence (verbatim from paper)

> We evaluate methods on two instruction-guided image-editing benchmarks: MagicBrush (MagBr) [32] and ZONE [11]. The former contains multi-turn edits where multiple instructions are used to edit one image iteratively, as opposed to single-turn edit where the image is edited once. We compare results quantitatively with the following metrics used in prior work [1, 11, 16, 32]: CLIP-T = sim(CLIPvis(I^e)', CLIPtxt(p^e)), CLIP-I = sim(CLIPvis((Ie)'), CLIPvis(Ie)), DINO-I = sim(DINOv2((Ie)'),DINOv2(Ie)) ... However, we note that as shown in Fig. 5 and in the supplemental material, these metrics do not always match the visual quality of the image edits, but are presented here for completeness.

## Citation

```bibtex
@misc{chen2025instructclip,
  title={Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2503.18406}
}
```

- arXiv: 2503.18406