Vtc Bench Eval

---
name: vtc-bench-eval
description: Evaluates multimodal large language models' ability to perform agentic visual reasoning by composing multiple OpenCV-based tool calls. It probes long-horizon planning, precise tool selection, and the capacity to chain coarse- to fine-grained visual operations to solve complex multi-step problems. Use when the user wants to benchmark on VTC-Bench, or asks about evaluating this task. Reports Average Pass Rate (APR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15030
  bibtex_key: zhu2026vtcbench
  confidence: high
---

# vtc-bench-eval

> VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining — Zhu et al. (2026) (arXiv:2603.15030, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to perform agentic visual reasoning by composing multiple OpenCV-based tool calls. It probes long-horizon planning, precise tool selection, and the capacity to chain coarse- to fine-grained visual operations to solve complex multi-step problems.

## Datasets

- **VTC-Bench** — total 680; splits: test (680)

## Metrics

- `Average Pass Rate (APR)` **(primary)** — range: percent
  - The percentage of benchmark problems for which the model produces the correct final answer within the maximum allowed interaction rounds. Calculated as (number of correctly solved problems / total problems) × 100.
- `Tool Call Rate (TCR)` — range: percent
  - The percentage of problems where the model successfully invokes at least one tool.
- `Tool Usage Efficiency (Eff.)` — range: percent
  - The ratio of effective toolchain steps to total attempted tool steps, measuring how closely the model's execution path matches the ground-truth trajectory.

## Input / output format

**Input**: A visual input (image) paired with a natural language task prompt describing the required visual reasoning or operation.

**Output**: A final textual or numerical answer generated after an iterative loop of tool calls and execution results, constrained by a maximum round limit.

## Scoring recipe

```python
def compute_apr(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if deterministic_match(pred, gold):
            correct += 1
        elif llm_judge_match(pred, gold, model='gpt-4o'):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often achieve high pass rates but use excessive redundant tool calls, failing to align with optimal ground-truth execution paths.
- Agents frequently over-rely on unverified intermediate tool outputs, bypassing critical cross-verification with the original visual input.
- Interface-based and code-based invocation yield different performance profiles, making direct comparison without controlling for the execution paradigm misleading.

## Evidence (verbatim from paper)

> The left panel of Fig.[5] reveals a positive correlation between Tool Call Rate (TCR) and Average Pass Rate (APR), validating the utility of active tool invocation. Conversely, the efficiency analysis in the right panel indicates that general-purpose models achieve superior efficiency even with lower TCR.

## Citation

```bibtex
@misc{zhu2026vtcbench,
  title={VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2603.15030}
}
```

- arXiv: 2603.15030