Deepgen 1.0 Eval

---
name: deepgen-1.0-eval
description: Evaluates a unified multimodal model's capabilities in text-to-image generation, image editing, and world-knowledge reasoning. It probes semantic alignment, long-horizon instruction following, fine-grained attribute binding, and precise text rendering across diverse scenarios. Use when the user wants to benchmark on GenEval, DPG-Bench, UniGenBench, WISE, T2I-CoREBench, ImgEdit, GEdit-EN, UniREditBench, RISE, CVTG-2K, or asks about evaluating this task. Reports GenEval.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12205
  bibtex_key: wang2026deepgen
  confidence: high
---

# deepgen-1.0-eval

> DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing — Wang et al. (2026) (arXiv:2602.12205, 2026)

## What this evaluates

Evaluates a unified multimodal model's capabilities in text-to-image generation, image editing, and world-knowledge reasoning. It probes semantic alignment, long-horizon instruction following, fine-grained attribute binding, and precise text rendering across diverse scenarios.

## Datasets

- **GenEval** — total ?; splits: test (-1)
- **DPG-Bench** — total ?; splits: test (-1)
- **UniGenBench** — total ?; splits: test (-1)
- **WISE** — total 1000; splits: test (1000)
- **T2I-CoREBench** — total ?; splits: test (-1)
- **ImgEdit** — total ?; splits: test (-1)
- **GEdit-EN** — total ?; splits: test (-1)
- **UniREditBench** — total 2700; splits: test (2700)
- **RISE** — total 327; splits: test (327)
- **CVTG-2K** — total 2000; splits: test (2000)

## Metrics

- `GenEval` **(primary)** — range: [0, 1]
  - Automated semantic alignment score computed using the official GenEval LLM-based evaluation script. Measures how well generated images match the semantic content of text prompts.
- `DPG-Bench` — range: other
  - Instruction-following score based on LLM-as-a-judge evaluations of long-prompt adherence and image quality.
- `UniGenBench` — range: other
  - Comprehensive fine-grained generation score aggregating performance across ten categories including attribute binding, style control, and text rendering.
- `WISE` — range: [0, 1]
  - World-knowledge reasoning generation score evaluating cultural, scientific, and spatial-temporal understanding.
- `T2I-CoREBench` — range: other
  - Reasoning generation score across eight structured categories (Logical, Behavioral, Hypothetical, etc.).
- `GEdit-EN` — range: other
  - Image editing score measuring instruction following, editing consistency, and output quality.
- `RISE` — range: other
  - Reasoning-based editing score across temporal, causal, spatial, and logical dimensions.
- `UniREditBench` — range: other
  - World-knowledge-grounded editing score covering real- and game-world scenarios.
- `CVTG-2K Word Accuracy` — range: [0, 1]
  - Character-level correctness and legibility score for English text generation in diverse scenarios.

## Input / output format

**Input**: Text prompt for generation tasks; text prompt plus reference image for editing tasks.

**Output**: A single generated image per prompt.

## Scoring recipe

```python
def evaluate(model, benchmark_name, test_set):
    scores = []
    for item in test_set:
        prompt = item['prompt']
        ref_img = item.get('reference_image')
        gen_img = model.generate(prompt, ref_img)
        score = get_benchmark_score(benchmark_name, gen_img, prompt, ref_img)
        scores.append(score)
    return sum(scores) / len(scores) if scores else 0.0
```

## Common pitfalls

- Different sampling schedules or CFG scales drastically change scores on GenEval and DPGBench.
- Using unofficial or outdated versions of benchmark evaluators (e.g., GenEval's LLM-based script) leads to non-comparable results.
- Confusing SFT-only and RL-finetuned checkpoint results, as RL consistently yields higher scores across all benchmarks.

## Evidence (verbatim from paper)

> We assess general text-to-image generation using GenEval [48] to measure fundamental semantic alignment, and DPG-Bench [49] to assess long-prompt instruction following. In addition, we adopt UniGenBench [27] for a comprehensive and fine-grained evaluation of general generation capability, covering ten major categories (e.g., attribute binding, style control, and text rendering).

## Citation

```bibtex
@misc{wang2026deepgen,
  title={DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.12205}
}
```

- arXiv: 2602.12205