Dreamomni2 Eval

---
name: dreamomni2-eval
description: Evaluates a model's ability to perform multimodal instruction-based image editing and generation, specifically testing adherence to text instructions while manipulating concrete objects and abstract attributes (e.g., texture, style) using multiple reference images. Use when the user wants to benchmark on DreamOmni2 benchmark, or asks about evaluating this task. Reports success editing ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06679
  bibtex_key: xia2025dreamomni2
  confidence: high
---

# dreamomni2-eval

> DreamOmni2: Multimodal Instruction-based Editing and Generation — Bin Xia et al. (arXiv:2510.06679, 2025)

## What this evaluates

Evaluates a model's ability to perform multimodal instruction-based image editing and generation, specifically testing adherence to text instructions while manipulating concrete objects and abstract attributes (e.g., texture, style) using multiple reference images.

## Datasets

- **DreamOmni2 benchmark** — total ?; splits: test (-1); repo https://github.com/dvlab-research/DreamOmni2

## Metrics

- `success editing ratio` **(primary)** — range: [0, 1]
  - The proportion of successfully edited or generated images out of the total test cases. Calculated as (number of successful cases) / (total cases). Evaluated separately for concrete objects and abstract attributions using VLM judges (Gemini 2.5, Doubao 1.6) and human experts.

## Input / output format

**Input**: A set of reference images (one or multiple) and a natural language instruction specifying the desired edit or generation (e.g., changing an object's attribute, adding an object, or modifying style).

**Output**: A single edited or generated image matching the instruction.

## Scoring recipe

```python
def calculate_success_rate(predictions, instructions, evaluator):
    success_count = 0
    for pred_img, instruction in zip(predictions, instructions):
        # Evaluator (VLM or human) checks if pred_img matches instruction
        # and maintains consistency with reference images
        is_success = evaluator.evaluate(pred_img, instruction)
        if is_success:
            success_count += 1
    return success_count / len(predictions)
```

## Common pitfalls

- VLM evaluators may struggle with abstract attributes (e.g., texture, style) and can introduce visual biases (e.g., color shifts like yellowing).
- The metric is a binary success ratio rather than a continuous quality score, so it does not capture partial compliance or subtle aesthetic degradation.
- Human evaluation is used as a proxy for ground truth, but inter-rater agreement and specific rubric details are not provided.

## Evidence (verbatim from paper)

> The models were evaluated for success rates by Gemini 2.5*(Google, [2025a])* and Doubao 1.6*(ByteDance, [2025])*, and several professional engineers manually assessed the results. Table 3: Quantitative comparison of multimodal instruction-based generation. We use Gemini*(Google, [2025a])* and Doubao*(ByteDance, [2025])* to evaluate the success editing ratio on concrete objects and abstract attributions, respectively. In addition, “Human” refers to professional engineers assessing the editing success rates of all models.

## Citation

```bibtex
@misc{xia2025dreamomni2,
  title={DreamOmni2: Multimodal Instruction-based Editing and Generation},
  author={Bin Xia et al.},
  year={2025},
  note={arXiv:2510.06679}
}
```

- arXiv: 2510.06679