Mobile O Eval

---
name: mobile-o-eval
description: Evaluates a compact on-device unified vision-language-diffusion model's capabilities in multimodal understanding, text-to-image generation, and image editing. It probes the model's ability to align cross-modal representations and generate high-fidelity images while maintaining real-time inference speeds on edge hardware. Use when the user wants to benchmark on GenEval, MMMU, MM-Vet, SEED, TextVQA, ChartQA, POPE, GQA, ImageEdit, or asks about evaluating this task. Reports GenEval overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20161
  bibtex_key: shaker2026mobileo
  confidence: high
---

# mobile-o-eval

> Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device — Shaker et al. (2026) (arXiv:2602.20161, 2026)

## What this evaluates

Evaluates a compact on-device unified vision-language-diffusion model's capabilities in multimodal understanding, text-to-image generation, and image editing. It probes the model's ability to align cross-modal representations and generate high-fidelity images while maintaining real-time inference speeds on edge hardware.

## Datasets

- **GenEval** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **ImageEdit** — total ?; splits: test (-1)

## Metrics

- `GenEval overall score` **(primary)** — range: [0, 1]
  - Average accuracy across six categories: Single Object, Two Object, Counting, Colors, Position, and Color Attributes. Scores range from 0 to 1 per category, averaged for the final score.
- `Understanding accuracy (%)` — range: percent
  - Average accuracy across seven benchmarks (MMMU, MM-Vet, SEED, TextVQA, ChartQA, POPE, GQA). Reported as a percentage.
- `ImageEdit score` — range: other
  - Composite score measuring edit fidelity and scene preservation on the ImageEdit benchmark.
- `Inference latency` — range: other
  - Time in milliseconds or seconds to run the vision encoder, compute time-to-first-token (TTFT), and generate an image (20 denoising steps) on specified edge devices.

## Input / output format

**Input**: For understanding: image (resized to 1024×1024) + text prompt/question. For generation: text prompt. For editing: source image + textual editing instruction.

**Output**: For understanding: text answer. For generation: 512×512 image. For editing: edited 512×512 image.

## Scoring recipe

```python
def compute_gen_eval_score(predictions, gold):
    categories = ['Single Obj.', 'Two Obj.', 'Counting', 'Colors', 'Position', 'Color Attri.']
    cat_scores = [sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) for cat in categories]
    return sum(cat_scores) / len(cat_scores)

def compute_understanding_accuracy(predictions, gold):
    accuracies = []
    for bench_preds, bench_gold in zip(predictions, gold):
        accuracies.append(sum(1 for p, g in zip(bench_preds, bench_gold) if p == g) / len(bench_gold))
    return sum(accuracies) / len(accuracies) * 100
```

## Common pitfalls

- Parameter counts reported in tables include all components (vision encoder, language model, diffusion/UNet), not just the LLM.
- Understanding images are resized to 1024×1024 via bicubic interpolation, while generation operates at 512×512; mixing resolutions will skew results.
- GenEval evaluation strictly uses raw prompts without any post-processing or prompt engineering.

## Evidence (verbatim from paper)

> Generation quality is measured by GenEval overall score. The proposed post-training stage consistently improves both capabilities.

## Citation

```bibtex
@misc{shaker2026mobileo,
  title={Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device},
  author={Shaker et al. (2026)},
  year={2026},
  note={arXiv:2602.20161}
}
```

- arXiv: 2602.20161