Prometheus Vision Eval

---
name: prometheus-vision-eval
description: Evaluates the fine-grained judgment capability of vision-language models by scoring generated text outputs against instance-specific rubrics and reference answers. It measures alignment with human preferences and state-of-the-art VLM judges across instruction following, VQA, and captioning tasks. Use when the user wants to benchmark on LLaVA-Bench, VisIT-Bench, Perception-Bench, OKVQA, VQAv2, TextVQA, COCO-Captions, NoCaps, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.06591
  bibtex_key: lee2024prometheusvision
  confidence: high
---

# prometheus-vision-eval

> Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation — Lee et al. (2024) (arXiv:2401.06591, 2024)

## What this evaluates

Evaluates the fine-grained judgment capability of vision-language models by scoring generated text outputs against instance-specific rubrics and reference answers. It measures alignment with human preferences and state-of-the-art VLM judges across instruction following, VQA, and captioning tasks.

## Datasets

- **LLaVA-Bench** — total 60; splits: test (60)
- **VisIT-Bench** — total 500; splits: test (500)
- **Perception-Bench** — total 500; splits: test (500)
- **OKVQA** — total 500; splits: test (500)
- **VQAv2** — total 500; splits: test (500)
- **TextVQA** — total 500; splits: test (500)
- **COCO-Captions** — total 500; splits: test (500)
- **NoCaps** — total 500; splits: test (500)

## Metrics

- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between the evaluator model's scores and the reference scores (human or GPT-4V). Calculated as the covariance of the two variables divided by the product of their standard deviations.
- `Kendall-Tau correlation` — range: [-1, 1]
  - Measures the ordinal association between two ranked lists of scores. Counts concordant and discordant pairs to assess ranking agreement.
- `Spearman correlation` — range: [-1, 1]
  - Measures the monotonic relationship between two ranked score lists. Computed as the Pearson correlation between the rank variables.
- `Pairwise Preference Win-rate` — range: [0, 1]
  - The percentage of instances where the evaluator model's generated feedback is preferred over a baseline model's feedback in direct human pairwise comparisons.

## Input / output format

**Input**: Image, text instruction or question, reference answer (for VQA/captioning tasks), and a fine-grained, instance-specific score rubric.

**Output**: A numerical score decision and accompanying language feedback explaining the score.

## Scoring recipe

```python
def compute_metrics(model_scores, ref_scores, pairwise_feedbacks):
    pearson = pearsonr(model_scores, ref_scores)
    kendall = kendalltau(model_scores, ref_scores)
    spearman = spearmanr(model_scores, ref_scores)
    
    wins = sum(1 for m, b in pairwise_feedbacks if human_prefers(m, b))
    win_rate = wins / len(pairwise_feedbacks)
    return pearson, kendall, spearman, win_rate
```

## Common pitfalls

- Language model baselines cannot process images directly; they require a separate captioning step via LLaVA-1.5, which may lose visual details critical for scoring.
- Rubrics are instance-specific and generated differently for human vs. GPT-4V evaluation setups, making cross-setup score comparisons invalid without normalization.
- VQA and captioning benchmarks use short ground-truth answers, while instruction-following benchmarks use long-form outputs, requiring distinct rubric generation and scoring strategies.

## Evidence (verbatim from paper)

> Then, we measure the correlation of the scoring decision by employing Pearson, Kendall-Tau, and Spearman as our metrics. Next, we ask human annotators to compare 2 language feedbacks that are sampled from either GPT-4, GPT-4V, or Prometheus-Vision (13B) and choose which one is better. Then, we measure the Pairwise Preference Win-rate between the 3 candidates.

## Citation

```bibtex
@misc{lee2024prometheusvision,
  title={Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2401.06591}
}
```

- arXiv: 2401.06591