Seed X Eval

---
name: seed-x-eval
description: Evaluates a multimodal model's ability to understand images and text (comprehension) and generate images from text instructions (generation). It probes fine-grained visual perception, reasoning, and compositional image synthesis. Use when the user wants to benchmark on VQAv2, GQA, POPE, MME, SEED, MMB, MM-Vet, MMMU, GenEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.14396
  bibtex_key: ge2024seedx
  confidence: high
---

# seed-x-eval

> SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation — Ge et al. (2024) (arXiv:2404.14396, 2024)

## What this evaluates

Evaluates a multimodal model's ability to understand images and text (comprehension) and generate images from text instructions (generation). It probes fine-grained visual perception, reasoning, and compositional image synthesis.

## Datasets

- **VQAv2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **MMB** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **GenEval** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard accuracy or benchmark-specific scoring metric used across all listed benchmarks. For GenEval, it measures the percentage of generated images that correctly satisfy all compositional constraints in the prompt.

## Input / output format

**Input**: Image and text prompt (for comprehension benchmarks), or text prompt only (for GenEval generation benchmark).

**Output**: Text answer (for comprehension), or generated image (for GenEval).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if evaluate_match(pred, gold_item):
            correct += 1
    return (correct / len(gold)) * 100

def evaluate_match(pred, gold):
    # For comprehension: exact string match or normalized answer match
    # For GenEval: strict attribute-level verification of generated image against prompt constraints
    return pred == gold
```

## Common pitfalls

- Benchmarks like MME and MM-Vet contain diverse sub-tasks (e.g., perception, reasoning, OCR) that should be reported separately rather than averaged into a single score.
- GenEval uses strict attribute-level verification; missing a single constraint (e.g., color, position, or count) causes the entire instance to be marked incorrect.
- Models may rely on dataset memorization or shortcut heuristics rather than true visual reasoning, especially on VQAv2 and GQA.

## Evidence (verbatim from paper)

> We evaluate the multimodal comprehension capabilities of SEED-X-I on widely recognized image-based vision-language benchmarks, which include VQAv2 [43], GQA [44], POPE [45], MME [46], SEED [47], MMB [48], MM-Vet [49], and MMMU [50]. As listed in Tab. 2, SEED-X-I achieves competitive performance across various benchmarks, even when compared to MLLMs specifically designed for multimodal comprehension. Image Generation. We evaluate the image generation capabilities of SEED-X-I on GenEval [51], which is a challenging benchmark to evaluate compositional image properties such as object cooccurrence, position, count, and color. As shown in Tab. 3, SEED-X obtains 51% overall accuracy, demonstrating the model's excellent instruction-following capabilities for image generation.

## Citation

```bibtex
@misc{ge2024seedx,
  title={SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation},
  author={Ge et al. (2024)},
  year={2024},
  note={arXiv:2404.14396}
}
```

- arXiv: 2404.14396