Mobile Gui Agent Eval

---
name: mobile-gui-agent-eval
description: Evaluates the capability of multimodal large language models to act as autonomous mobile GUI agents. It probes their ability to plan tasks, predict correct interaction types, accurately ground UI elements, and successfully complete complex, multi-step workflows in both static and dynamic Android environments. Use when the user wants to benchmark on AndroidControl, AndroidLab, Android Agent Arena (A3), or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21496
  bibtex_key: xiao2025uigenie
  confidence: high
---

# mobile-gui-agent-eval

> UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents — Han Xiao et al. (arXiv:2505.21496, 2025)

## What this evaluates

Evaluates the capability of multimodal large language models to act as autonomous mobile GUI agents. It probes their ability to plan tasks, predict correct interaction types, accurately ground UI elements, and successfully complete complex, multi-step workflows in both static and dynamic Android environments.

## Datasets

- **AndroidControl** — total ?; splits: test (-1)
- **AndroidLab** — total 138; splits: test (138)
- **Android Agent Arena (A3)** — total 201; splits: test (201)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Measures the percentage of successfully completed tasks or actions. Calculated as the number of successful tasks divided by the total number of tasks.
- `Type Accuracy` — range: percent
  - Assesses the agent’s ability to correctly predict action types such as click, long-press, or scroll. Calculated as the number of correct action type predictions divided by total predictions.
- `Grounding Accuracy` — range: percent
  - Evaluates the precision in locating and interacting with the correct UI elements. Calculated as the number of correctly grounded interactions divided by total interactions.
- `Essential State Achievement Rate (ESAR)` — range: percent
  - Measures the rate at which agents achieve critical intermediate states required for task completion, evaluated via function-based or LLM-based essential state checks.

## Input / output format

**Input**: Current and historical screenshots, task instructions/goals, and action history. In dynamic settings, the model receives live emulator state updates after each action.

**Output**: Predicted action type (e.g., click, scroll, long-press) and UI element grounding coordinates/identifiers.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    type_correct = sum(1 for p, g in zip(predictions, golds) if p.action_type == g.action_type)
    grounding_correct = sum(1 for p, g in zip(predictions, golds) if p.grounding == g.grounding)
    task_success = sum(1 for p, g in zip(predictions, golds) if p.task_completed == g.task_completed)
    n = len(predictions)
    return {
        'Type Accuracy': type_correct / n,
        'Grounding Accuracy': grounding_correct / n,
        'Success Rate': task_success / n
    }
```

## Common pitfalls

- Static vs. dynamic evaluation protocols differ significantly; static benchmarks only compare predicted actions against ground truth without executing them, while dynamic benchmarks run actions in an emulator, making success dependent on state transitions.
- Grounding Accuracy is frequently reported as 0.0 for proprietary models (e.g., Claude, GPT-4o) in Table 2, likely because they do not output explicit UI element coordinates/IDs required for this metric.
- Task complexity categories (easy/medium/hard) are strictly defined by step count (<5, 5-10, >10) and must be used when reporting stratified performance.

## Evidence (verbatim from paper)

> For both settings, we employ three primary metrics: (1) Success Rate (SR), measuring the percentage of successfully completed actions; (2) Type Accuracy, assessing the agent’s ability to correctly predict action types (e.g., click, long-press, scroll); and (3) Grounding Accuracy, evaluating the agent’s precision in locating and interacting with the correct UI elements.

## Citation

```bibtex
@misc{xiao2025uigenie,
  title={UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents},
  author={Han Xiao et al.},
  year={2025},
  note={arXiv:2505.21496}
}
```

- arXiv: 2505.21496