Osworld Eval

---
name: osworld-eval
description: Evaluates multimodal agents' ability to perform open-ended, real-world desktop computer tasks across multiple operating systems. It probes GUI grounding, multi-app workflow navigation, and executable action prediction in dynamic, interactive environments. Use when the user wants to benchmark on OSWorld, or asks about evaluating this task. Reports success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.07972
  bibtex_key: xie2024osworld
  confidence: high
---

# osworld-eval

> OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Xie et al. (2024) (arXiv:2404.07972, 2024)

## What this evaluates

Evaluates multimodal agents' ability to perform open-ended, real-world desktop computer tasks across multiple operating systems. It probes GUI grounding, multi-app workflow navigation, and executable action prediction in dynamic, interactive environments.

## Datasets

- **OSWorld** — total 369; splits: test (-1)

## Metrics

- `success` **(primary)** — range: percent
  - Calculated as the number of tasks where the agent successfully completes the instruction (returns DONE and passes execution verification) divided by the total number of tasks. Evaluated via execution-based scripts.

## Input / output format

**Input**: Task instruction, current screen screenshot, and optionally an accessibility tree or Set-of-Mark annotated image.

**Output**: Python code block using `pyautogui` for mouse/keyboard actions, or special tokens: `WAIT`, `FAIL`, or `DONE`.

## Scoring recipe

```python
def compute_success(predictions, gold_tasks):
    completed = 0
    for task in gold_tasks:
        actions = predictions[task.id]
        env = launch_sandbox(task.os)
        result = run_agent(actions, env, max_steps=15, max_time=1800)
        if result.status == "DONE" and verify_goal(task.goal, result.state):
            completed += 1
    return completed / len(gold_tasks) * 100
```

## Common pitfalls

- Agents must not use `pyautogui.LocationCenterOnScreen` or `pyautogui.Screen()`; coordinates must be predicted manually.
- Evaluation is strictly bounded by 15 interaction steps and a 30-minute time limit per task to prevent infinite loops.
- Accessibility tree inputs require platform-specific filtering (Ubuntu vs Windows) to stay within token limits.

## Evidence (verbatim from paper)

> State-of-the-art LLM/VLM agents achieve only 12.24% success—far below human performance (72.36%)—highlighting critical deficiencies in GUI grounding, operational knowledge, and action prediction, especially in complex, dynamic workflows.

## Citation

```bibtex
@misc{xie2024osworld,
  title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments},
  author={Xie et al. (2024)},
  year={2024},
  note={arXiv:2404.07972}
}
```

- arXiv: 2404.07972