Llava Bench Eval

---
name: llava-bench-eval
description: Assesses multimodal chatbot capabilities, including conversation, detailed description, and complex visual reasoning. It measures how well a model follows instructions and understands novel or challenging visual inputs compared to a strong text-only baseline. Use when the user wants to benchmark on LLaVA-Bench, or asks about evaluating this task. Reports relative_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.08485
  bibtex_key: liu2023visualinstructiontuning
  confidence: high
---

# llava-bench-eval

> Visual Instruction Tuning — Haotian Liu et al. (2023) (arXiv:2304.08485, 2023)

## What this evaluates

Assesses multimodal chatbot capabilities, including conversation, detailed description, and complex visual reasoning. It measures how well a model follows instructions and understands novel or challenging visual inputs compared to a strong text-only baseline.

## Datasets

- **LLaVA-Bench** — total 150; splits: COCO (90), In-the-Wild (60)

## Metrics

- `relative_score` **(primary)** — range: percent
  - GPT-4 judges model responses on a 1-10 scale for helpfulness, relevance, accuracy, and detail. Scores are reported as a percentage relative to a text-only GPT-4 baseline that uses ground-truth image captions as visual input.

## Input / output format

**Input**: Image and a textual question (conversation, detailed description, or complex reasoning).

**Output**: Free-form textual response.

## Scoring recipe

```python
baseline_resp = GPT4(question, ground_truth_caption)
model_resp = Model(image, question)
score_1_10 = GPT4_Judge(question, ground_truth_caption, baseline_resp, model_resp)
relative_score = (score_1_10 / 10) * 100
```

## Common pitfalls

- The baseline uses ground-truth captions, giving it an unfair advantage over models that must generate descriptions from scratch.
- LLM-as-a-judge scores can vary with temperature; the paper notes evaluating GPT-4 three times for consistency.
- Benchmarks use out-of-domain or highly challenging images, making direct comparison with standard datasets like COCO difficult.

## Evidence (verbatim from paper)

> To provide an approximate theoretical upper bound, we create a reference prediction based on the question and the ground-truth textual descriptions, using the text-only GPT-4. ... It evaluates the helpfulness, relevance, accuracy, and level of detail of the responses from the assistants, and gives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. ... We report relative scores w.r.t. the text-only GPT-4 model that uses the textual ground truth description as visual input.

## Citation

```bibtex
@misc{liu2023visualinstructiontuning,
  title={Visual Instruction Tuning},
  author={Haotian Liu et al. (2023)},
  year={2023},
  note={arXiv:2304.08485}
}
```

- arXiv: 2304.08485