Real World Cross App Eval

---
name: real-world-cross-app-eval
description: Assesses multi-step agent capabilities including tool use, GUI grounding, compositional generalization, and long-horizon planning across real-world desktop and web applications. It evaluates whether agents can execute complex, cross-application workflows and self-evaluate their trajectories. Use when the user wants to benchmark on Real-World Cross-Application Benchmark Suite, or asks about evaluating this task. Reports Success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17918
  bibtex_key: zheng2024agentstudio
  confidence: high
---

# real-world-cross-app-eval

> AgentStudio: A Toolkit for Building General Virtual Agents — Longtao Zheng et al. (2024) (arXiv:2403.17918, 2024)

## What this evaluates

Assesses multi-step agent capabilities including tool use, GUI grounding, compositional generalization, and long-horizon planning across real-world desktop and web applications. It evaluates whether agents can execute complex, cross-application workflows and self-evaluate their trajectories.

## Datasets

- **Real-World Cross-Application Benchmark Suite** — total 77; splits: test (77)

## Metrics

- `Success` **(primary)** — range: percent
  - Percentage of tasks where the agent's trajectory successfully achieves the intended outcome as verified by an environment evaluation function $f_{\mathcal{E}}$.
- `Critic Accuracy` — range: percent
  - Accuracy of the model's self-evaluation in correctly judging whether its own trajectory succeeded or failed.

## Input / output format

**Input**: Natural language instruction $g$, optionally an environment reset function $f_{\mathcal{R}}$, and an environment evaluation function $f_{\mathcal{E}}$.

**Output**: A sequence of actions (API calls, GUI interactions, tool uses) forming a trajectory.

## Scoring recipe

```python
successes = 0
critic_correct = 0
for task in dataset:
    trajectory = model.execute(task.instruction)
    is_success = task.evaluate(trajectory)
    model_judgment = model.self_evaluate(trajectory)
    if is_success: successes += 1
    if model_judgment == is_success: critic_correct += 1
return (successes / len(dataset)) * 100, (critic_correct / len(dataset)) * 100
```

## Common pitfalls

- Static single-step datasets cannot evaluate multi-step tasks that allow multiple valid solutions.
- Level 3 tasks require human-in-the-loop evaluation because automatic rule-based evaluators are insufficient for complex cross-application workflows.

## Evidence (verbatim from paper)

> A task is formalized as a tuple: $T\=(g,f_{\mathcal{R}},f_{\mathcal{E}})$, where $g$ is a natural language description of the task instruction, $f_{\mathcal{R}}$ optionally resets the environment, and $f_{\mathcal{E}}$ optionally evaluates the outcome trajectory. ... Each task is evaluated based on two metrics: the percentage of tasks the model successfully completes (Success), and the accuracy in evaluating the success of trajectories (Critic Accuracy).

## Citation

```bibtex
@misc{zheng2024agentstudio,
  title={AgentStudio: A Toolkit for Building General Virtual Agents},
  author={Longtao Zheng et al. (2024)},
  year={2024},
  note={arXiv:2403.17918}
}
```

- arXiv: 2403.17918