Visit Bench Eval

---
name: visit-bench-eval
description: Evaluates vision-language models' ability to follow complex, real-world instructions on images. It probes open-ended generation, context-sensitive reasoning, and instruction-conditioned captioning by measuring how well model outputs align with human preferences and high-quality references. Use when the user wants to benchmark on VisIT-Bench, or asks about evaluating this task. Reports Elo rating, Win rate vs. reference.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.06595
  bibtex_key: [REDACTED]
  confidence: high
---

# visit-bench-eval

> VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use — Bitton et al. (2023) (arXiv:2308.06595, 2023)

## What this evaluates

Evaluates vision-language models' ability to follow complex, real-world instructions on images. It probes open-ended generation, context-sensitive reasoning, and instruction-conditioned captioning by measuring how well model outputs align with human preferences and high-quality references.

## Datasets

- **VisIT-Bench** — total 592; splits: test (592)

## Metrics

- `Elo rating` **(primary)** — range: unbounded (typically 700-1400)
  - Pairwise comparison metric where each human judgment is treated as a match. The difference in Elo ratings between two models estimates the win probability when pitting them against each other.
- `Win rate vs. reference` **(primary)** — range: percent
  - Percentage of forced-choice pairwise judgments where a model's output is preferred over a human-verified GPT-4 reference output. Calculated as (number of wins against reference) / (total comparisons against reference).

## Input / output format

**Input**: Image and a text instruction. For pairwise evaluation, the model receives the image, instruction, instruction-conditioned caption, and two candidate text responses (Response A and Response B).

**Output**: Text response generated by the vision-language model. For evaluation, a forced-choice preference (A or B) or a tie.

## Scoring recipe

```python
def compute_win_rate(predictions, golds):
    # predictions: list of model outputs, golds: list of reference outputs
    wins = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (wins / len(golds)) * 100
```

## Common pitfalls

- Forced-choice setup does not allow ties, forcing annotators/LLMs to arbitrarily pick a winner when responses are equal quality.
- GPT-4 evaluator outputs 'tie' in 10-15% of cases; the authors resolve this by random choice, which may introduce noise.
- Elo ratings are highly dependent on the specific set of models evaluated and match counts, limiting cross-benchmark comparability.

## Evidence (verbatim from paper)

> We summarize the results with two metrics: Relative metric: Elo We follow [26] and compute Elo ratings, treating each pairwise human judgement as a "match." The difference between the Elo ratings of two different models provides an estimate for the win probability when pitting model A vs. model B. More details are in Appendix D. Absolute metric: Win rate vs. reference. We provide a win-rate vs. the human-verified reference. We use the 1.4K pairwise human judgments where one of A or B is the reference. We report the percent of cases where the human judge prefers the output from that model vs. the human-verified GPT-4 reference output.

## Citation

```bibtex
@misc{bitton2023visitbench,
  title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use},
  author={Bitton et al. (2023)},
  year={2023},
  note={arXiv:2308.06595}
}
```

- arXiv: 2308.06595