Gui Ceval Eval

---
name: gui-ceval-eval
description: Evaluates multimodal large language models and agents on Chinese mobile GUI interaction tasks. It probes atomic capabilities like visual perception, grounding, and planning, as well as end-to-end execution reliability in both offline simulation and real-device online environments. Use when the user wants to benchmark on GUI-CEval, or asks about evaluating this task. Reports Online Agent success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15039
  bibtex_key: li2026guiceval
  confidence: high
---

# gui-ceval-eval

> GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents — Li et al. (2026) (arXiv:2603.15039, 2026)

## What this evaluates

Evaluates multimodal large language models and agents on Chinese mobile GUI interaction tasks. It probes atomic capabilities like visual perception, grounding, and planning, as well as end-to-end execution reliability in both offline simulation and real-device online environments.

## Datasets

- **GUI-CEval** — total ?; splits: test (-1)

## Metrics

- `Online Agent success rate` **(primary)** — range: percent
  - Percentage of tasks completed successfully on real mobile devices. Measures end-to-end capability under real-world constraints.
- `Step Success Rate (SSR)` — range: percent
  - Percentage of individual steps executed correctly in offline execution traces. Computed as correct steps divided by total ground-truth steps.
- `Grounding accuracy` — range: percent
  - Percentage of correctly localized click targets. Measured only on samples where both annotation and prediction are clicks to avoid perception bias from mixed action types.
- `Overall average score` — range: percent
  - Aggregated percentage across five hierarchical capability dimensions: perception, planning, reflection, execution, and evaluation.

## Input / output format

**Input**: Chinese language task instructions paired with mobile GUI screenshots (images). Screenshots may be provided at original resolution or scaled down to 1/2, 1/4, or 1/6 of original width/height.

**Output**: Model-generated action sequences (e.g., click coordinates, text input, scroll) and/or reasoning chains. Predictions are parsed to determine if the predicted action matches the ground-truth annotation for each step.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, online_tasks):
    # Grounding Accuracy (clicks only)
    click_pairs = [(p, g) for p, g in zip(predictions, ground_truth) if g.action == 'click']
    grounding_acc = sum(1 for p, g in click_pairs if p.action == 'click' and match_coords(p, g)) / len(click_pairs) * 100
    
    # Step Success Rate (SSR)
    ssr = sum(1 for p, g in zip(predictions, ground_truth) if actions_match(p, g)) / len(ground_truth) * 100
    
    # Online Agent Success Rate
    online_success = sum(1 for task in online_tasks if task.completed_on_device) / len(online_tasks) * 100
    
    return {'grounding_accuracy': grounding_acc, 'ssr': ssr, 'online_success_rate': online_success}
```

## Common pitfalls

- Grounding accuracy is strictly limited to click actions to avoid perception bias from mixed action types; evaluating it on all actions inflates or deflates the metric unfairly.
- Online success rate exhibits a cliff-like decline beyond 6 steps, making short-horizon or offline metrics poor proxies for real-world deployment capability.
- Initial page state heavily influences results; evaluating only from the home screen overestimates robustness for uncontrolled user paths and diverse app entry points.

## Evidence (verbatim from paper)

> When examined by capability dimension, perception-related scores such as Perception and Grounding remain high (peaking at 82.28% and 90.10%, respectively), while reasoning and correction-related metrics (Reflection and Evaluation) lag far behind (maximums of 21.01% and 49.24%). For Online Agent—the most challenging task involving real interactions—the best success rate is 33.33% (UI-TARS-72B-SFT), with over half of the models below 20%.

## Citation

```bibtex
@misc{li2026guiceval,
  title={GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.15039}
}
```

- arXiv: 2603.15039