Vtcbench Eval

---
name: vtcbench-eval
description: Evaluates the ability of visual token compression methods to retain task-relevant visual information across five dimensions: global understanding, spatial and counting, reasoning and common sense, style and emotion, and local details. Use when the user wants to benchmark on VTCBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.21232
  bibtex_key: li2026qmoP
  confidence: high
---

# vtcbench-eval

> QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression — Li et al. (2026) (arXiv:2603.21232, 2026)

## What this evaluates

Evaluates the ability of visual token compression methods to retain task-relevant visual information across five dimensions: global understanding, spatial and counting, reasoning and common sense, style and emotion, and local details.

## Datasets

- **VTCBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) × 100.

## Input / output format

**Input**: An image paired with a textual question or instruction.

**Output**: A generated text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Performance is highly sensitive to the number of retained visual tokens; comparisons must fix token counts (e.g., 144 vs 192) to ensure fairness.
- Different compression strategies inherently favor different task types (e.g., pruning for local details, pooling for global context), so reporting only a single average can mask trade-offs.

## Evidence (verbatim from paper)

> We evaluate our model on ten representative public vision-understanding benchmarks, as well as on our newly constructed VTCBench. ... Table 3 reports the accuracy of different methods, where a higher score indicates better retention of visual information.

## Citation

```bibtex
@misc{li2026qmoP,
  title={QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.21232}
}
```

- arXiv: 2603.21232