Osworld Verified Eval

---
name: osworld-verified-eval
description: Evaluates an agent's ability to autonomously plan and execute multi-step GUI automation tasks across various desktop applications. It measures the success rate on both in-distribution tasks from the OSWorld-Verified benchmark and out-of-distribution tasks across six distinct Linux applications. Use when the user wants to benchmark on OSWorld-Verified, OOD GUI Benchmark, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09662
  bibtex_key: [REDACTED]
  confidence: high
---

# osworld-verified-eval

> TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution — Deyang Jiang et al. (2026) (arXiv:2602.09662, 2026)

## What this evaluates

Evaluates an agent's ability to autonomously plan and execute multi-step GUI automation tasks across various desktop applications. It measures the success rate on both in-distribution tasks from the OSWorld-Verified benchmark and out-of-distribution tasks across six distinct Linux applications.

## Datasets

- **OSWorld-Verified** — total ?; splits: test (-1)
- **OOD GUI Benchmark** — total 120; splits: test (120)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Percentage of tasks completed successfully out of the total number of test tasks. Calculated as (number of successful tasks / total tasks) * 100.

## Input / output format

**Input**: Task description, initial GUI screenshot(s), and optionally official application documentation. The model receives visual states and textual instructions to generate GUI interaction actions.

**Output**: A sequence of GUI interaction actions (trajectory) executed to complete the task.

## Scoring recipe

```python
def calculate_success_rate(predictions, gold_tasks):
    successes = 0
    for pred, task in zip(predictions, gold_tasks):
        # GPT-4o evaluates trajectory against task description
        # Task description explicitly constrains intermediate nodes & final UI state
        eval1 = gpt4o_evaluate_trajectory(pred, task.description)
        eval2 = gpt4o_evaluate_trajectory(pred, task.description)
        if eval1.success and eval2.success:
            successes += 1
    return (successes / len(gold_tasks)) * 100
```

## Common pitfalls

- The OOD evaluation relies on GPT-4o as an automated judge, which may introduce LLM-as-a-judge biases or inconsistencies compared to ground-truth execution logs.
- Success requires passing two consecutive evaluation rounds, making the metric stricter than standard single-pass automated metrics.
- OSWorld-Verified results are sourced from external leaderboards or original papers rather than re-run evaluations, potentially introducing environment or version mismatches.

## Evidence (verbatim from paper)

> For evaluation, we employ GPT-4o to assess the complete interaction trajectory against the task description, deeming a task successful only if it passes two consecutive evaluation rounds to ensure robustness. Crucially, to mitigate ambiguity and enhance the accuracy of this model-based evaluation, we explicitly constrain critical intermediate nodes and the required final UI state within each task description. Table 4: Comparison of foundation models on our constructed OOD benchmark (Success Rate).

## Citation

```bibtex
@misc{jiang2026treecua,
  title={TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution},
  author={Deyang Jiang et al. (2026)},
  year={2026},
  note={arXiv:2602.09662}
}
```

- arXiv: 2602.09662