X Omni Eval

---
name: x-omni-eval
description: Evaluates the text rendering, text-to-image generation, and image understanding capabilities of a discrete autoregressive image generation model trained with reinforcement learning. It probes the model's ability to follow complex instructions, render long texts accurately, and generate high-fidelity images without relying on classifier-free guidance. Use when the user wants to benchmark on OneIG-Bench, LongText-Bench, DPG-Bench, GenEval, POPE, GQA, MMBench, SEEDBench-Img, DocVQA, OCRBench, or asks about evaluating this task. Reports DPG-Bench Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.22058
  bibtex_key: geng2025xomni
  confidence: high
---

# x-omni-eval

> X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again — Geng et al. (2025) (arXiv:2507.22058, 2025)

## What this evaluates

Evaluates the text rendering, text-to-image generation, and image understanding capabilities of a discrete autoregressive image generation model trained with reinforcement learning. It probes the model's ability to follow complex instructions, render long texts accurately, and generate high-fidelity images without relying on classifier-free guidance.

## Datasets

- **OneIG-Bench** — total ?; splits: test (-1)
- **LongText-Bench** — total 160; splits: test (160)
- **DPG-Bench** — total ?; splits: test (-1)
- **GenEval** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **SEEDBench-Img** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)

## Metrics

- `DPG-Bench Overall` **(primary)** — range: [0, 100]
  - Composite score averaging category-level scores (Global, Entity, Attribute, Relation, Other) evaluated by a vision-language model comparing generated images to prompts.
- `GenEval Overall` — range: [0, 1]
  - Composite score averaging category-level scores (Single, Two, Counting, Colors, Position, Color Attr.) evaluating precise attribute and spatial reasoning in generated images.
- `OneIG-Bench Text Rendering` — range: [0, 1]
  - Composite score derived from Edit Distance, Completion Rate, and Word Accuracy to assess text rendering proficiency in English and Chinese.
- `LongText-Bench` — range: [0, 1]
  - Score evaluating the capacity to precisely render long Chinese and English texts across 8 scenarios using 160 prompts.
- `POPE` — range: [0, 1]
  - Accuracy/F1 score for object hallucination detection in image understanding.
- `GQA` — range: [0, 1]
  - Accuracy score for visual question answering focusing on compositional reasoning.
- `MMBench` — range: [0, 1]
  - Accuracy score for multimodal benchmark evaluating general multimodal understanding.
- `SEEDBench-Img` — range: [0, 1]
  - Accuracy score for image understanding across multiple tasks.
- `DocVQA` — range: [0, 1]
  - Exact match accuracy for document visual question answering.
- `OCRBench` — range: other
  - OCR score measuring optical character recognition capability on images.

## Input / output format

**Input**: Text prompts for generation benchmarks; image-text pairs for understanding benchmarks.

**Output**: Generated images for generation tasks; text answers for understanding tasks.

## Scoring recipe

```python
def evaluate_generation(pred_images, prompts, benchmark):
    scores = []
    for img, prompt in zip(pred_images, prompts):
        if benchmark == 'GenEval':
            prompt = rewrite_prompt(prompt)
        scores.append(vlm_score(img, prompt))
    return mean(scores)

def evaluate_understanding(pred_answers, gold_answers, benchmark):
    if benchmark in ['POPE', 'GQA', 'MMBench', 'SEEDBench-Img', 'DocVQA']:
        return exact_match_or_f1(pred_answers, gold_answers)
    elif benchmark == 'OCRBench':
        return ocr_score(pred_answers, gold_answers)
    return 0.0
```

## Common pitfalls

- Prompt rewriting is required for GenEval evaluation; skipping it invalidates results.
- GPT-4o API rejects certain prompts, so reported results exclude these cases.
- Classifier-free guidance (CFG) significantly impacts autoregressive generation quality; results without CFG should be reported separately to avoid misleading comparisons.

## Evidence (verbatim from paper)

> We evaluate text-to-image generation on two widely recognized benchmarks: DPG-Bench*[hu2024dpgbench]* and GenEval*[ghosh2023geneval]*. Detailed results are shown in Table[2] and Table[3] respectively. Note that prompt rewriting is employed for GenEval evaluation.

## Citation

```bibtex
@misc{geng2025xomni,
  title={X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2507.22058}
}
```

- arXiv: 2507.22058