Gui Agent Halluc Eval

---
name: gui-agent-halluc-eval
description: This protocol evaluates GUI agents on visual grounding, action execution, and hallucination rates across mobile, desktop, and web interfaces. It measures how well models localize UI elements, execute multi-step tasks under varying instruction granularities, and avoid perception or reasoning errors. Use when the user wants to benchmark on ScreenSpot-V2, ScreenSpot-Pro, AndroidControl, GUI-Odyssey, or asks about evaluating this task. Reports Action Type Accuracy (Type), Grounding Accuracy (GR), Step-wise Success Rate (SR), Hallucination Rate (HR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17284
  bibtex_key: jin2026halluclear
  confidence: high
---

# gui-agent-halluc-eval

> HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents — Chao Jin et al. (2026) (arXiv:2604.17284, 2026)

## What this evaluates

This protocol evaluates GUI agents on visual grounding, action execution, and hallucination rates across mobile, desktop, and web interfaces. It measures how well models localize UI elements, execute multi-step tasks under varying instruction granularities, and avoid perception or reasoning errors.

## Datasets

- **ScreenSpot-V2** — total ?; splits: test (-1)
- **ScreenSpot-Pro** — total ?; splits: test (-1)
- **AndroidControl** — total ?; splits: test (-1)
- **GUI-Odyssey** — total ?; splits: test (-1)

## Metrics

- `Action Type Accuracy (Type)` **(primary)** — range: percent
  - Exact match accuracy between the predicted and ground-truth action types.
- `Grounding Accuracy (GR)` **(primary)** — range: percent
  - Evaluates the precise localization of click points within specific action categories (e.g., CLICK).
- `Step-wise Success Rate (SR)` **(primary)** — range: percent
  - Quantifies alignment between prediction and ground truth by calculating the intersection of Type and GR.
- `Hallucination Rate (HR)` **(primary)** — range: percent
  - Overall rate of hallucination behaviors classified by VLM judges, normalized by the judge's credibility score to mitigate bias.

## Input / output format

**Input**: GUI screenshot/image paired with a text instruction (global high-level or low-level with pre-annotated step-by-step actions).

**Output**: Predicted action type, grounding coordinates (click point), and reasoning/trajectory steps.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    type_acc = sum(1 for p, g in zip(predictions, golds) if p.type == g.type) / len(golds)
    gr_acc = sum(1 for p, g in zip(predictions, golds) if dist(p.click, g.click) <= THRESHOLD) / len(golds)
    sr = type_acc * gr_acc
    hr = 1.0 - (sum(1 for p in predictions if not p.hallucination) / len(predictions))
    return {'Type': type_acc, 'GR': gr_acc, 'SR': sr, 'HR': hr}
```

## Common pitfalls

- Low-level vs high-level difficulty settings differ in input granularity; confusing them leads to incorrect capability assessment.
- HR is normalized by VLM judge credibility scores; ignoring this calibration inflates or deflates hallucination rates depending on judge bias.
- GR requires a spatial threshold for click point localization that is not explicitly defined in the provided text, requiring external benchmark documentation.

## Evidence (verbatim from paper)

> Specifically, we employ three primary metrics: Action Type Accuracy (Type), which measures the exact match accuracy between the predicted and ground-truth action types; Grounding Accuracy (GR), which evaluates the precise localization of click points within specific action categories (e.g., CLICK); and Step-wise Success Rate (SR), which quantifies the alignment between prediction and ground truth by calculating the intersection of Type and GR.

## Citation

```bibtex
@misc{jin2026halluclear,
  title={HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents},
  author={Chao Jin et al. (2026)},
  year={2026},
  note={arXiv:2604.17284}
}
```

- arXiv: 2604.17284