Gedit Imgedit Bench Eval

---
name: gedit-imgedit-bench-eval
description: Evaluates the quality of AI-generated image edits by measuring semantic consistency, perceptual quality, and overall fidelity against source images and edit instructions. Use when the user wants to benchmark on GEdit-Bench, ImgEdit-Bench, or asks about evaluating this task. Reports Overall (GEdit-Bench).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.14119
  bibtex_key: [REDACTED]
  confidence: high
---

# gedit-imgedit-bench-eval

> NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining — Kuprashevich et al. (2025) (arXiv:2507.14119, 2025)

## What this evaluates

Evaluates the quality of AI-generated image edits by measuring semantic consistency, perceptual quality, and overall fidelity against source images and edit instructions.

## Datasets

- **GEdit-Bench** — total ?; splits: test (-1)
- **ImgEdit-Bench** — total ?; splits: test (-1)

## Metrics

- `Semantic Consistency (SC)` — range: [0, 10]
  - LLM-as-a-judge score (GPT-4o) measuring how well the edited image preserves the original semantic content relative to the source image and edit prompt.
- `Perceptual Quality (PQ)` — range: [0, 10]
  - LLM-as-a-judge score (GPT-4o) measuring the visual fidelity and aesthetic quality of the edited image.
- `Overall (GEdit-Bench)` **(primary)** — range: [0, 10]
  - LLM-as-a-judge score (GPT-4o) aggregating semantic consistency and perceptual quality into a single overall edit quality rating.
- `Overall (ImgEdit-Bench)` — range: [1, 5]
  - LLM-as-a-judge score (GPT-4o) averaging multiple edit quality criteria, each rated on a 1-to-5 scale.

## Input / output format

**Input**: Source image and edit instruction/prompt (implied by image editing task; model generates an edited image).

**Output**: Edited image, evaluated by GPT-4o against source image and prompt to produce criterion scores.

## Scoring recipe

```python
def compute_metrics(predictions, gold, benchmark):
    scores = []
    for pred_img, src_img, prompt in zip(predictions, gold['src'], gold['prompt']):
        if benchmark == 'GEdit-Bench':
            sc = llm_judge(pred_img, src_img, prompt, 'Semantic Consistency', scale=10)
            pq = llm_judge(pred_img, src_img, prompt, 'Perceptual Quality', scale=10)
            ov = llm_judge(pred_img, src_img, prompt, 'Overall', scale=10)
            scores.append((sc, pq, ov))
        elif benchmark == 'ImgEdit-Bench':
            crit_scores = [llm_judge(pred_img, src_img, prompt, c, scale=5) for c in criteria]
            scores.append(mean(crit_scores))
    return mean(scores), std(scores), ci95(scores)
```

## Common pitfalls

- LLM-as-a-judge scoring is sensitive to prompt wording and model version; must strictly use the official VIEScore and ImgEdit-Bench evaluation prompts.
- Benchmarks use different rating scales (0–10 vs. 1–5), so scores cannot be directly compared across datasets without normalization.
- Reported results average over 3 inference seeds; single-seed evaluations may not capture model variance or confidence intervals.

## Evidence (verbatim from paper)

> For GEdit-Bench, we use the VIEScore setup with GPT-4o to report Semantic Consistency (SC, 0-10), Perceptual Quality (PQ, 0-10), and Overall (O). For the ImgEdit-Bench evaluation, we adopt the original authors’ protocol: GPT-4o is used to score edited images across several criteria, each rated on a 1-to-5 scale.

## Citation

```bibtex
@misc{kuprashevich2025nohumansrequired,
  title={NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining},
  author={Kuprashevich et al. (2025)},
  year={2025},
  note={arXiv:2507.14119}
}
```

- arXiv: 2507.14119