Gui Agent Kv Eval

---
name: gui-agent-kv-eval
description: Evaluates the accuracy and efficiency of GUI agents under varying KV cache compression budgets across visual grounding, offline action prediction, and online task completion benchmarks. Use when the user wants to benchmark on ScreenSpotV2, ScreenSpot-Pro, AndroidControl, Multimodal-Mind2Web, AgentNetBench, OSWorld-Verified, or asks about evaluating this task. Reports step accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.00536
  bibtex_key: huang2025guikv
  confidence: high
---

# gui-agent-kv-eval

> GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness — Huang et al. (2025) (arXiv:2510.00536, 2025)

## What this evaluates

Evaluates the accuracy and efficiency of GUI agents under varying KV cache compression budgets across visual grounding, offline action prediction, and online task completion benchmarks.

## Datasets

- **ScreenSpotV2** — total ?; splits: test (-1)
- **ScreenSpot-Pro** — total ?; splits: test (-1)
- **AndroidControl** — total ?; splits: test (-1)
- **Multimodal-Mind2Web** — total ?; splits: test (-1)
- **AgentNetBench** — total ?; splits: test (-1)
- **OSWorld-Verified** — total ?; splits: test (-1)

## Metrics

- `click accuracy` — range: [0, 1]
  - Proportion of test samples where the model's predicted coordinate falls within the ground truth bounding box.
- `success rate` — range: [0, 1]
  - Proportion of tasks where a series of operations are successfully executed and specific milestones are achieved.
- `step accuracy` **(primary)** — range: [0, 1]
  - Proportion of test samples where a single predicted step contains the correct operation (e.g., click, write) and arguments (e.g., coordinate or textual content).

## Input / output format

**Input**: GUI screenshots (current and recent history frames) paired with natural language instructions or task descriptions.

**Output**: Predicted UI element coordinates (bounding boxes) or discrete action steps with arguments (e.g., click, write, coordinates).

## Scoring recipe

```python
def compute_metric(predictions, golds, metric_type):
    if metric_type == 'click_accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p_in_box(p, g)) / len(golds)
    elif metric_type == 'success_rate':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif metric_type == 'step_accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p.op == g.op and p.args == g.args) / len(golds)
```

## Common pitfalls

- Metrics are benchmark-specific (click accuracy vs success rate vs step accuracy), so direct numerical comparison across different datasets is invalid.
- KV cache budgets are evaluated at discrete, non-linear levels (1%, 3%, 5%, 10%, 15%, 20%, 40%, 80%, 100%), and performance does not always scale monotonically with budget.
- Efficiency metrics (MFLOPs per decoded token) are only reported for AgentNetBench, making cross-dataset efficiency comparisons impossible.

## Evidence (verbatim from paper)

> For ScreenSpot-V2 and ScreenSpot-Pro, we measure click accuracy, defined as the proportion of test samples where the model’s predicted coordinate falls within the ground truth bounding box. For OSWorld-Verified, we use success rate, determined by whether a series of operations are successfully executed and specific milestones are achieved. For the remaining benchmarks, we adopt step accuracy, which assesses whether a single predicted step contains the correct operation (e.g., click, write) and arguments (e.g., coordinate or textual content).

## Citation

```bibtex
@misc{huang2025guikv,
  title={GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2510.00536}
}
```

- arXiv: 2510.00536