Computer Use Agent Eval

---
name: computer-use-agent-eval
description: Evaluates computer-use agents on desktop task completion in online and offline settings, and measures the precision of a video-to-action module in detecting GUI events and extracting interaction parameters from screen recordings. Use when the user wants to benchmark on OSWorld-Verified, AgentNetBench, Video2Action Held-out Test Set, or asks about evaluating this task. Reports task success rate, step success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19488
  bibtex_key: lu2025videoagenttrek
  confidence: high
---

# computer-use-agent-eval

> VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos — Lu et al. (2025) (arXiv:2510.19488, 2025)

## What this evaluates

Evaluates computer-use agents on desktop task completion in online and offline settings, and measures the precision of a video-to-action module in detecting GUI events and extracting interaction parameters from screen recordings.

## Datasets

- **OSWorld-Verified** — total 369; splits: test (369)
- **AgentNetBench** — total 100; splits: test (100)
- **Video2Action Held-out Test Set** — total 20282; splits: test (20282)

## Metrics

- `task success rate` **(primary)** — range: percent
  - Percentage of tasks completed successfully out of the total number of tasks in the benchmark.
- `step success rate` **(primary)** — range: percent
  - Percentage of individual interaction steps executed correctly out of the total steps in the benchmark.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall for action event detection. A prediction counts as a hit if its type matches and its temporal interval has any overlap with a ground-truth event.
- `Accuracy` — range: [0, 1]
  - Proportion of manually validated action parameter predictions that correctly explain the observed on-screen transition between pre/post frames.

## Input / output format

**Input**: For agent benchmarks: OS state/screenshots and task instructions. For Video2Action: Screen-capture video frames.

**Output**: For agent benchmarks: Sequence of GUI actions (click, drag, press, scroll, type) with parameters (coordinates, text). For Video2Action: Action event tuples (type, start_time, end_time) and interaction parameters.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_f1(preds, gold):
    hits = 0
    for p in preds:
        if any(p['type'] == g['type'] and intervals_overlap(p['t_s'], p['t_e'], g['t_s'], g['t_e']) for g in gold):
            hits += 1
    precision = hits / len(preds) if preds else 0
    recall = hits / len(gold) if gold else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Confusing online (OSWorld-Verified) vs offline (AgentNetBench) evaluation settings, which use different environments and action budgets.
- Assuming temporal overlap requires exact boundary matching; the protocol uses any overlap for event detection hits.
- Parameter accuracy is manually assessed due to missing target-element boxes, not automatically computed.

## Evidence (verbatim from paper)

> On OSWorld-Verified, our complete approach achieves a task success rate of 14.13%, demonstrating a 4.83 percentage point improvement (+52% relative) over SFT-only training (9.3%) and more than tripling the performance of the base model (4.5%). On AgentNetBench, incorporating VideoAgentTrek pretraining achieves a step success rate of 69.3%, representing a 5.2 percentage point improvement over the SFT-only baseline (64.1%) and a substantial 30.8 percentage point gain over the base model (38.5%). A prediction counts as a hit iff its type matches and its interval has any temporal overlap with a ground-truth event; unmatched predictions are false positives and unmatched ground truths are false negatives. We report per-type Precision/Recall/F1 and micro/macro aggregates.

## Citation

```bibtex
@misc{lu2025videoagenttrek,
  title={VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2510.19488}
}
```

- arXiv: 2510.19488