Mmbench Gui Eval

---
name: mmbench-gui-eval
description: Evaluates GUI agents across four hierarchical levels: content understanding, element grounding, single-app task automation, and multi-app task collaboration. It probes visual grounding, cross-platform generalization, and long-horizon planning capabilities while measuring both task success and step efficiency. Use when the user wants to benchmark on MMBench-GUI, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.19478
  bibtex_key: wang2025mmbenchgui
  confidence: high
---

# mmbench-gui-eval

> MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents — Xuehui Wang et al. (arXiv:2507.19478, 2025)

## What this evaluates

Evaluates GUI agents across four hierarchical levels: content understanding, element grounding, single-app task automation, and multi-app task collaboration. It probes visual grounding, cross-platform generalization, and long-horizon planning capabilities while measuring both task success and step efficiency.

## Datasets

- **MMBench-GUI** — total ?; splits: test (-1); repo https://github.com/open-compass/MMBench-GUI

## Metrics

- `exact-match accuracy` **(primary)** — range: percent
  - Returns 1 if the model's parsed output exactly matches the gold answer, else 0. Scores are aggregated as a weighted average across platforms based on task proportions.
- `success_rate (SR)` — range: percent
  - Returns 1 if the agent's final GUI state matches the predefined success criteria, else 0. Averaged across all tasks and platforms.
- `EQA` — range: other
  - Efficiency–Quality-Aware metric that jointly measures task accuracy and step efficiency. It rewards successful task completion while penalizing redundant or inefficient action sequences.

## Input / output format

**Input**: GUI screenshot paired with a task description or instruction/question. Auxiliary artifacts such as accessibility (A11y) trees and Set-of-Marks (SoM) data are deliberately omitted.

**Output**: L1/L2: Natural language answer or element identifier. L3/L4: Sequence of GUI actions/steps leading to a final state.

## Scoring recipe

```python
def score_l1_l2(pred, gold):
    parsed = parse_with_regex(pred)
    return 1.0 if parsed == gold else 0.0

def score_l3_l4(pred_actions, gold_state, max_steps=50):
    success = check_final_state(pred_actions, gold_state)
    if success:
        eqa = 1.0 / len(pred_actions)  # Penalizes longer sequences
        return {"SR": 1.0, "EQA": eqa}
    return {"SR": 0.0, "EQA": 0.0}

# Aggregate across tasks/platforms using weighted averages
```

## Common pitfalls

- Output parsing for L1/L2 is highly sensitive to model instruction-following variability, often requiring custom regex-based parsers to extract valid answers.
- L3/L4 evaluation only checks the final state against success criteria, ignoring intermediate steps and masking inefficient or redundant action sequences.
- Platform scores must be weighted by task proportion before aggregation; simple averaging skews results due to uneven task distribution across OS/browser environments.

## Evidence (verbatim from paper)

> Model outputs were assessed using exact-match evaluation protocol, analogous to standard practices in grounding and QA tasks. However, given the variability in instruction-following abilities across different models, for example, the QA tasks in L1, we observed that some model outputs could not be reliably parsed. To address this, we implemented a hybrid parsing mechanism based on multiple regular expressions to robustly extract valid answers.

## Citation

```bibtex
@misc{wang2025mmbenchgui,
  title={MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents},
  author={Xuehui Wang et al.},
  year={2025},
  note={arXiv:2507.19478}
}
```

- arXiv: 2507.19478