Groundnext Eval

---
name: groundnext-eval
description: Evaluates vision-language models on UI element localization and grounding across desktop, mobile, and web interfaces. It measures how accurately a model can identify and locate specific UI components based on text instructions, and assesses their effectiveness in multi-step agentic tasks. Use when the user wants to benchmark on SSPro, OSW-G, MMB-GUI, SSv2, UI-V, OSWorld-Verified, or asks about evaluating this task. Reports average performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.07332
  bibtex_key: feizi2025grounding
  confidence: high
---

# groundnext-eval

> Grounding Computer Use Agents on Human Demonstrations — Feizi et al. (2025) (arXiv:2511.07332, 2025)

## What this evaluates

Evaluates vision-language models on UI element localization and grounding across desktop, mobile, and web interfaces. It measures how accurately a model can identify and locate specific UI components based on text instructions, and assesses their effectiveness in multi-step agentic tasks.

## Datasets

- **SSPro** — total ?; splits: test (-1)
- **OSW-G** — total ?; splits: test (-1)
- **MMB-GUI** — total ?; splits: test (-1)
- **SSv2** — total ?; splits: test (-1)
- **UI-V** — total ?; splits: test (-1)
- **OSWorld-Verified** — total 361; splits: test (361)

## Metrics

- `average performance` **(primary)** — range: percent
  - Arithmetic mean of accuracy scores across the listed benchmarks (SSPro, OSW-G, MMB-GUI, SSv2, UI-V). Reported as a percentage.

## Input / output format

**Input**: Screen capture (screenshot) of a desktop, mobile, or web interface paired with a natural language instruction describing the target UI element or task.

**Output**: Bounding box coordinates or UI element identifier corresponding to the target described in the instruction.

## Scoring recipe

```python
def compute_avg_accuracy(predictions, golds):
    correct = 0
    total = 0
    for pred, gold in zip(predictions, golds):
        if match(pred, gold): # IoU > threshold or exact element match
            correct += 1
        total += 1
    return (correct / total) * 100
```

## Common pitfalls

- The 'average performance' metric calculation varies across tables; Figure 3 explicitly excludes UI-Vision, while Table 2 includes it in the average.
- Agentic evaluation (OSWorld-Verified) relies on a specific external planner (o3) and a fixed Ubuntu 1920x1080 Docker environment, limiting direct comparability with models evaluated in different OS/resolution settings.
- RL post-training yields only marginal gains (~1-2 points) over high-quality SFT, so reporting RL-only results without SFT baselines can be misleading.

## Evidence (verbatim from paper)

> Our models achieve the highest average performance for both $3$B and $7$B model sizes.

## Citation

```bibtex
@misc{feizi2025grounding,
  title={Grounding Computer Use Agents on Human Demonstrations},
  author={Feizi et al. (2025)},
  year={2025},
  note={arXiv:2511.07332}
}
```

- arXiv: 2511.07332