Medebench Eval

---
name: medebench-eval
description: Evaluates the reliability and clinical appropriateness of text-guided medical image editing models. It probes anatomical localization precision, preservation of surrounding clinical context, and overall visual realism across diverse medical imaging modalities and anatomical regions. Use when the user wants to benchmark on MedEBench, or asks about evaluating this task. Reports GPT-4o Editing Accuracy, Masked SSIM, GPT-4o Visual Quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01921
  bibtex_key: liu2025medebench
  confidence: high
---

# medebench-eval

> MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing — Liu et al. (2025) (arXiv:2506.01921, 2025)

## What this evaluates

Evaluates the reliability and clinical appropriateness of text-guided medical image editing models. It probes anatomical localization precision, preservation of surrounding clinical context, and overall visual realism across diverse medical imaging modalities and anatomical regions.

## Datasets

- **MedEBench** — total 1182; splits: test (-1)

## Metrics

- `GPT-4o Editing Accuracy` **(primary)** — range: [0, 1]
  - LLM-based scoring where GPT-4o evaluates whether the edited image correctly implements the text instruction relative to the original. Scores are scaled from 0–10 to 0–1.
- `Masked SSIM` **(primary)** — range: [0, 1]
  - Structural Similarity Index computed only on the region-of-interest (ROI) mask to measure how well the surrounding clinical context is preserved during editing. Scaled to [0, 1].
- `GPT-4o Visual Quality` **(primary)** — range: [0, 1]
  - LLM-based scoring where GPT-4o rates the photorealism and clinical plausibility of the edited image. Scores are scaled from 0–10 to 0–1.
- `FID` — range: other
  - Fréchet Inception Distance measuring the distributional distance between real and generated medical images in a feature space. Lower values indicate better distributional match.

## Input / output format

**Input**: A medical image (e.g., X-ray, CT, dermoscopy) paired with a natural language editing instruction (e.g., 'remove polyp', 'straighten spine').

**Output**: A single edited medical image generated by the model.

## Scoring recipe

```python
def evaluate_model(model, dataset):
    ea_scores, cp_scores, vq_scores = [], [], []
    for img, prompt, roi_mask in dataset:
        edited = model.generate(img, prompt)
        # Context Preservation
        cp = masked_ssim(img, edited, roi_mask)
        # Editing Accuracy & Visual Quality (LLM-based)
        ea = normalize_llm_score(gpt4o.evaluate(edited, img, prompt, expected_change), 0, 10)
        vq = normalize_llm_score(gpt4o.evaluate(edited, prompt, quality_prompt), 0, 10)
        ea_scores.append(ea)
        cp_scores.append(cp)
        vq_scores.append(vq)
    return {
        'EA': mean(ea_scores),
        'CP': mean(cp_scores),
        'VQ': mean(vq_scores)
    }
```

## Common pitfalls

- FID is computed at the distribution level and cannot be used for instance-level correlation with human ratings or per-sample metric reporting.
- Models often trade off visual quality for context preservation; high VQ scores may indicate over-editing or repainting of large regions rather than precise localized changes.
- In-context learning with few-shot demonstrations frequently degrades performance on fine-grained medical edits due to model confusion between examples and test inputs.

## Evidence (verbatim from paper)

> EA = GPT-4o Editing Accuracy_Desc, CP = Masked SSIM, VQ = GPT-4o Visual Quality_Desc. EA and VQ scaled from 0–10 to 0–1.

## Citation

```bibtex
@misc{liu2025medebench,
  title={MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.01921}
}
```

- arXiv: 2506.01921