Crop And Zoom Tool Use Eval

---
name: crop-and-zoom-tool-use-eval
description: Evaluates vision-language models' ability to use a crop-and-zoom tool for high-resolution visual question answering, disentangling intrinsic capability improvements from tool-induced gains and harms across multiple benchmarks. Use when the user wants to benchmark on VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Harm, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.01334
  bibtex_key: ma2026visiontoolusereinforcement
  confidence: high
---

# crop-and-zoom-tool-use-eval

> What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom — Yan Ma et al. (2026) (arXiv:2602.01334, 2026)

## What this evaluates

Evaluates vision-language models' ability to use a crop-and-zoom tool for high-resolution visual question answering, disentangling intrinsic capability improvements from tool-induced gains and harms across multiple benchmarks.

## Datasets

- **VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Harm** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Binary correctness of the final textual answer against the ground truth. Computed separately for tool-available (Acc_w) and tool-free (Acc_wo) protocols.
- `tool_induced_gap` — range: [-1, 1]
  - Difference between tool-available and tool-free accuracy (G(t) = Acc_w - Acc_wo), decomposed into Gross Gain (Call Gain + Schema Gain) and Gross Harm (Call Harm + Schema Harm).

## Input / output format

**Input**: An image and a natural language question.

**Output**: A final textual answer to the question.

## Scoring recipe

```python
# Compute baseline and tool-assisted accuracy
acc_wo = mean(1 for pred, gold in zip(predictions_tool_free, golds) if pred == gold)
acc_w  = mean(1 for pred, gold in zip(predictions_tool_available, golds) if pred == gold)
# Compute tool-induced gap and decomposition
G = acc_w - acc_wo
# Factorize into Mass, Policy, Quality for specific subsets (fail/succ)
# Alignment: intersection of human and AI judgments on Call Gain samples
```

## Common pitfalls

- Using Acc_schema (schema provided but execution forbidden) as the baseline instead of Acc_wo artificially inflates tool gains by treating recovery from an artificial constraint as improvement.
- Evaluating call quality P(✓|c, D_fail) on the dynamic failure set at each checkpoint confuses execution quality improvements with the shifting difficulty of the remaining failures.
- Assuming tool-use RL primarily optimizes tool handling; the protocol shows intrinsic capability drift dominates performance gains.

## Evidence (verbatim from paper)

> We evaluate models under both tool-available and tool-free protocols on six benchmarks (VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Harm), with greedy decoding (temperature = 0) at regular checkpoints throughout training. The absolute accuracy for both Acc_w (solid line) and Acc_wo (dashed line) increases monotonically.

## Citation

```bibtex
@misc{ma2026visiontoolusereinforcement,
  title={What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom},
  author={Yan Ma et al. (2026)},
  year={2026},
  note={arXiv:2602.01334}
}
```

- arXiv: 2602.01334