Pira Bench Eval

---
name: pira-bench-eval
description: This benchmark evaluates multimodal large language models on proactive intent recommendation from continuous, noisy GUI visual streams. It probes the model's ability to track long-horizon user behavior, distinguish true latent goals from background noise, and exercise operational restraint by remaining silent when no action is required. Use when the user wants to benchmark on PIRA-Bench, or asks about evaluating this task. Reports S_final.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.08013
  bibtex_key: [REDACTED]
  confidence: high
---

# pira-bench-eval

> PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents — Chai et al. (2026) (arXiv:2603.08013, 2026)

## What this evaluates

This benchmark evaluates multimodal large language models on proactive intent recommendation from continuous, noisy GUI visual streams. It probes the model's ability to track long-horizon user behavior, distinguish true latent goals from background noise, and exercise operational restraint by remaining silent when no action is required.

## Datasets

- **PIRA-Bench** — total 100; splits: test (100)

## Metrics

- `Precision` — range: percent
  - True positives divided by all positive predictions. Measures the accuracy of intent suggestions.
- `Recall` — range: percent
  - True positives divided by all actual positive intents. Measures the model's sensitivity to user goals.
- `F1_avg` — range: percent
  - Harmonic mean of Precision and Recall. Balances accuracy and sensitivity for intent detection.
- `FPS_norm` — range: percent
  - Normalized False Positive Score measuring noise robustness. Penalizes hallucinations during idle or noisy frames.
- `S_final` **(primary)** — range: percent
  - Final composite score calculated as the product of F1_avg and FPS_norm. Rewards both intent detection capability and operational restraint.

## Input / output format

**Input**: A sequence of N=10 resized GUI screenshots per turn, accompanied by the user's profile/context. The model receives these frames incrementally over a trajectory.

**Output**: The model must output a prediction per turn: either a specific actionable intent, or a 'no action' (noise) designation. In the PIRF setting, it also updates a memory state (CREATE, RESUME, UPDATE, or IDLE).

## Scoring recipe

```python
# predictions: list of model outputs per frame/turn
# gold: list of ground truth labels per frame/turn
tp = sum(1 for p, g in zip(predictions, gold) if p == g and p != "noise")
fp = sum(1 for p, g in zip(predictions, gold) if p != g and p != "noise")
fn = sum(1 for p, g in zip(predictions, gold) if p == "noise" and g != "noise")
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1_avg = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
# FPS_norm approximates normalized false positive rate on noise frames
fps_norm = 1.0 - (fp / max(1, total_noise_frames))
s_final = f1_avg * fps_norm
return precision, recall, f1_avg, fps_norm, s_final
```

## Common pitfalls

- High recall alone is misleading; models with high recall but low precision suffer from 'over-proactivity' and receive heavily penalized final scores due to hallucinations during idle moments.
- Visual noise drastically collapses precision across all models, even when recall remains stable or slightly increases, indicating a failure to exercise operational restraint in cluttered streams.
- Human baseline performance is significantly higher but requires 15-20x more inference time, meaning the benchmark metrics do not account for latency or real-time responsiveness.

## Evidence (verbatim from paper)

> The performance of GPT-5.2 in the Naive setting serves as a stark illustration of the "over-proactivity" trap. While it achieves a remarkable recall of 83.37%, which is the highest among all naive baselines, this sensitivity proves deceptive. The accompanying precision is critically low (31.95%), and its noise robustness score (FPS_norm) is the worst in the table at 31.31%.

## Citation

```bibtex
@misc{chai2026pirabench,
  title={PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents},
  author={Chai et al. (2026)},
  year={2026},
  note={arXiv:2603.08013}
}
```

- arXiv: 2603.08013