Uinnexus Eval

---
name: uinnexus-eval
description: Evaluates mobile agents' ability to complete long-horizon, dependency-rich tasks on real mobile applications. It specifically probes atomic-to-compositional generalization, testing how well agents handle task concatenation, context transitions, and deep analysis across different app types and languages. Use when the user wants to benchmark on UI-NEXUS, or asks about evaluating this task. Reports Success Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08972
  bibtex_key: guo2025uinnexus
  confidence: high
---

# uinnexus-eval

> Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System — Guo et al. (2025) (arXiv:2506.08972, 2025)

## What this evaluates

Evaluates mobile agents' ability to complete long-horizon, dependency-rich tasks on real mobile applications. It specifically probes atomic-to-compositional generalization, testing how well agents handle task concatenation, context transitions, and deep analysis across different app types and languages.

## Datasets

- **UI-NEXUS** — total 50; splits: test (50)

## Metrics

- `Success Rate` **(primary)** — range: percent
  - Percentage of tasks completed successfully out of the total number of tasks. Calculated as (number of successful completions / total tasks) * 100.
- `PGR` — range: percent
  - Performance Gap Recovered. Measures how much of the gap between weak (direct compositional execution) and strong ceiling (oracle decomposition) performance is closed. Formula: (Atomic-to-compositional performance - Weak performance) / (Strong ceiling performance - Weak performance).
- `Inference Latency` — range: other
  - Average time taken per agent step to process UI state and output an action.
- `Inference Cost` — range: other
  - Average monetary cost in USD incurred per agent step.

## Input / output format

**Input**: UI state (screenshots and/or accessibility trees) and natural language task instructions (either atomic subtasks or compositional long-horizon instructions).

**Output**: Sequential mobile UI actions (e.g., click, swipe, type, back) to interact with the application and complete the given task.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    successful = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successful / len(gold)) * 100

def compute_pgr(weak_perf, strong_perf, oracle_perf):
    if oracle_perf == weak_perf:
        return 0.0
    return ((strong_perf - weak_perf) / (oracle_perf - weak_perf)) * 100
```

## Common pitfalls

- Agents frequently fail due to 'Budget Exceeded' or 'Premature' termination rather than actual task failure, especially for agent-as-a-model baselines.
- Chinese online service apps often lack proper accessibility tree support, which artificially penalizes agents that rely on structured UI parsing (e.g., M3A).
- Success Rate is measured end-to-end; partial completion or getting stuck in loops counts as failure, making the metric strict for long-horizon tasks.

## Evidence (verbatim from paper)

> We evaluate representative mobile agents implemented as agentic workflow and agent-as-a-model. ... UI-NEXUS poses substantial challenges on all five mobile agent baselines, with no agent exceeding task success rate of more than 50% on all subsets. ... Inspired by Weak-to-Strong Generalization [64], we define the Performance Gap Recovered (PGR) as: PGR = (Atomic-to-compositional performance - Weak performance) / (Strong ceiling performance - Weak performance)

## Citation

```bibtex
@misc{guo2025uinnexus,
  title={Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2506.08972}
}
```

- arXiv: 2506.08972