Androidlh Eval

---
name: androidlh-eval
description: Evaluates a GUI agent's ability to perform long-horizon, multi-app tasks in a mobile environment. It probes the agent's planning and skill-retrieval capabilities across complex, real-world application scenarios. Use when the user wants to benchmark on AndroidLH, or asks about evaluating this task. Reports task success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10387
  bibtex_key: xie2025mirage1
  confidence: high
---

# androidlh-eval

> Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills — Xie et al. (2025) (arXiv:2506.10387, 2025)

## What this evaluates

Evaluates a GUI agent's ability to perform long-horizon, multi-app tasks in a mobile environment. It probes the agent's planning and skill-retrieval capabilities across complex, real-world application scenarios.

## Datasets

- **AndroidLH** — total 30; splits: test (30)

## Metrics

- `task success rate` **(primary)** — range: percent
  - Percentage of long-horizon tasks completed successfully.
- `completion rate` — range: percent
  - Proportion of sub-goals or steps completed successfully across tasks.

## Input / output format

**Input**: Screen screenshots of the Android emulator environment for multi-app, long-horizon tasks.

**Output**: Action selection to navigate and interact across multiple applications.

## Scoring recipe

```python
success = 1 if task goal is met according to system state-based verification, else 0.
SR = mean(success) * 100.
CR = mean(completion proportion) * 100.
```

## Common pitfalls

- Tasks are synthetically generated by GPT-4o, which may not perfectly reflect real-world complexity or edge cases.
- System state-based verification requires accurate state tracking across app boundaries, which can be brittle if the emulator state diverges from the agent's perception.

## Evidence (verbatim from paper)

> Specifically, Mirage-1 exhibits significant performance gains with mean improvements of 53.3% in completion rate and 56.1% in success rate compared to the baseline agents.

## Citation

```bibtex
@misc{xie2025mirage1,
  title={Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.10387}
}
```

- arXiv: 2506.10387