Mobile Eval

---
name: mobile-eval
description: Evaluates an autonomous mobile device agent's ability to execute multi-step UI operations using visual perception. It probes task planning, self-reflection, and cross-application interaction under varying instruction complexity. Use when the user wants to benchmark on Mobile-Eval, or asks about evaluating this task. Reports Success (Su).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.16158
  bibtex_key: [REDACTED]
  confidence: high
---

# mobile-eval

> Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception — Wang et al. (2024) (arXiv:2401.16158, 2024)

## What this evaluates

Evaluates an autonomous mobile device agent's ability to execute multi-step UI operations using visual perception. It probes task planning, self-reflection, and cross-application interaction under varying instruction complexity.

## Datasets

- **Mobile-Eval** — total 30; splits: test (30)

## Metrics

- `Success (Su)` **(primary)** — range: [0, 1]
  - Boolean; 1 if the agent successfully completes the user instruction, 0 otherwise.
- `Process Score (PS)` — range: [0, 1]
  - Number of correct execution steps divided by the total number of steps taken by the agent.
- `Relative Efficiency (RE)` — range: other
  - Reported as a ratio of agent steps to human baseline steps (agent_steps / human_steps). Human steps are manually recorded as the optimal solution.
- `Completion Rate (CR)` — range: [0, 1]
  - Number of human-optimal steps successfully executed by the agent divided by the total number of human-optimal steps.

## Input / output format

**Input**: Natural language instruction paired with real-time mobile screen screenshots (visual perception).

**Output**: Sequence of discrete UI operations (e.g., click, type, swipe, navigate) executed on the device.

## Scoring recipe

```python
def compute_metrics(agent_actions, human_actions, success_flag):
    su = 1.0 if success_flag else 0.0
    ps = count_correct_steps(agent_actions) / len(agent_actions) if len(agent_actions) > 0 else 0.0
    re = f"{len(agent_actions)} / {len(human_actions)}"
    cr = count_human_steps_completed(agent_actions) / len(human_actions) if len(human_actions) > 0 else 0.0
    return su, ps, re, cr
```

## Common pitfalls

- Process Score (PS) can remain high even when the final task fails, as it only measures intermediate step accuracy rather than end-to-end success.
- Relative Efficiency (RE) is reported as a raw step ratio rather than a normalized percentage, which can exceed 1.0 if the agent takes more steps than the human baseline.
- Completion Rate (CR) is capped at 1.0 and does not penalize the agent for taking extra or redundant steps beyond the human baseline.

## Evidence (verbatim from paper)

> Metrics. We have designed four metrics to assess the performance of the Mobile-Agent from different perspectives:

* •

    Success (Su): If the Mobile-Agent completes the instruction, it is considered successful.

* •

    Process Score (PS): This metric measures the accuracy of each step in the execution of instructions. Specifically, it equals the number of correct steps divided by the total number of steps.

## Citation

```bibtex
@misc{wang2024mobileagent,
  title={Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.16158}
}
```

- arXiv: 2401.16158