State Control Eval

---
name: state-control-eval
description: Evaluates multimodal agents' ability to perceive current GUI states from screenshots, interpret natural language toggle instructions, and execute precise click actions. It specifically probes state-aware reasoning by measuring accuracy on both positive and negative toggle instructions, as well as grounding precision and false positive/negative rates. Use when the user wants to benchmark on state control benchmark, dynamic evaluation benchmark, or asks about evaluating this task. Reports O-AMR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.13615
  bibtex_key: wu2025star
  confidence: high
---

# state-control-eval

> See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles — Wu et al. (2025) (arXiv:2509.13615, 2025)

## What this evaluates

Evaluates multimodal agents' ability to perceive current GUI states from screenshots, interpret natural language toggle instructions, and execute precise click actions. It specifically probes state-aware reasoning by measuring accuracy on both positive and negative toggle instructions, as well as grounding precision and false positive/negative rates.

## Datasets

- **state control benchmark** — total ?; splits: test (-1); repo https://github.com/ZrW00/StaR
- **dynamic evaluation benchmark** — total 20; splits: test (20); repo https://github.com/ZrW00/StaR

## Metrics

- `O-AMR` **(primary)** — range: percent
  - Overall Action Match Rate: proportion of test samples where the predicted action matches the ground truth in both type and parameters (e.g., click coordinates, text content, app names).
- `N-AMR` — range: percent
  - Negative Action Match Rate: proportion of samples with negative instructions where the predicted action correctly matches the ground truth in type and parameters.
- `TMR` — range: percent
  - Type Match Rate: proportion of test samples where the predicted action type matches the ground truth.
- `Task Success Rate` — range: [0, 1]
  - Proportion of successful task trajectories. For dynamic tasks with multiple subtasks, success is partial: if half succeed and half fail, the rate is 0.5.

## Input / output format

**Input**: Current GUI screenshot(s), natural language instruction (specifying desired toggle state), and optionally prior action history or low-level step instructions.

**Output**: A reasoning process (Thought) followed by an action decision (Action) specifying action type and parameters (e.g., normalized click coordinates [0,1000], text content, or app name).

## Scoring recipe

```python
def compute_amr(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred['type'] == gold['type'] and pred['params'] == gold['params']:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Confusing positive-instruction metrics (P-AMR, P-FNR) with negative-instruction metrics (N-AMR, N-FPTR, N-FPR), which require opposite state-change logic.
- Forgetting that click coordinates must be normalized to [0, 1000] during evaluation, as specified in the implementation details.
- Misinterpreting partial success in dynamic environments: a task with multiple subtasks scores 0.5 if exactly half succeed, not 0 or 1.

## Evidence (verbatim from paper)

> Type Match Rate (TMR)↑: Proportion of test samples where the predicted action type matches the ground truth. Action Match Rate (AMR)↑: Proportion of test samples where the predicted action matches the ground truth in both type and parameters (e.g., click coordinates, text content, app names).

## Citation

```bibtex
@misc{wu2025star,
  title={See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.13615}
}
```

- arXiv: 2509.13615