Worldgui Eval

---
name: worldgui-eval
description: Evaluates an agent's ability to automate desktop and web GUI tasks from arbitrary starting states. It probes robustness to dynamic initial conditions, contextual variations, and multi-step interaction planning in real-world software environments. Use when the user wants to benchmark on WorldGUI, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.08047
  bibtex_key: zhao2025worldgui
  confidence: high
---

# worldgui-eval

> WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point — Zhao et al. (2025) (arXiv:2502.08047, 2025)

## What this evaluates

Evaluates an agent's ability to automate desktop and web GUI tasks from arbitrary starting states. It probes robustness to dynamic initial conditions, contextual variations, and multi-step interaction planning in real-world software environments.

## Datasets

- **WorldGUI** — total 611; splits: test (611); repo https://github.com/showlab/WorldGUI

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - The percentage of tasks successfully completed by the agent out of the total number of tasks. Calculated as (number of successful tasks / total tasks) × 100.

## Input / output format

**Input**: Screenshot of the current GUI state (1920×1080), task instruction/query, and extracted GUI element information (positions, text via OCR/parser).

**Output**: Sequential computer-use actions (e.g., mouse clicks, keyboard inputs) specified by coordinates or element identifiers to execute the task.

## Scoring recipe

```python
def compute_sr(predictions, gold_tasks):
    success_count = 0
    for pred, task in zip(predictions, gold_tasks):
        if pred == 'success':  # task completion verified by environment
            success_count += 1
    return (success_count / len(gold_tasks)) * 100
```

## Common pitfalls

- Meta tasks (standard instructions) vs. Augmented tasks (interface/context variations) show large performance gaps; comparing only overall SR masks this robustness deficit.
- Experiments limit agent trials to 4×N+1 per task to control costs, which artificially caps success rates and may not reflect true convergence on complex tasks.
- Human baseline performance assumes experts watched an instructional video only once, a constraint that differs from standard zero-shot or few-shot agent evaluations.

## Evidence (verbatim from paper)

> Following the previous works of OSworld and AssistGUI, we use Success Rate (SR) as the metric. Table 4 reports the success rates (SR) of different agents and human experts on our WorldGUI benchmark, broken down by task type (Meta vs. Aug.) across five categories: Office, Win. Usage, Web, Coding, and Media.

## Citation

```bibtex
@misc{zhao2025worldgui,
  title={WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2502.08047}
}
```

- arXiv: 2502.08047