Vl Rewardbench Eval

---
name: vl-rewardbench-eval
description: Evaluates vision-language generative reward models (VL-GenRMs) on their ability to judge multimodal response preferences. It specifically probes visual perception, reasoning, and hallucination detection by presenting models with image-text queries and paired candidate responses. Use when the user wants to benchmark on VL-RewardBench, or asks about evaluating this task. Reports Overall Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.17451
  bibtex_key: li2024vlrewardbench
  confidence: high
---

# vl-rewardbench-eval

> VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models — Lei Li et al. (2024) (arXiv:2411.17451, 2024)

## What this evaluates

Evaluates vision-language generative reward models (VL-GenRMs) on their ability to judge multimodal response preferences. It specifically probes visual perception, reasoning, and hallucination detection by presenting models with image-text queries and paired candidate responses.

## Datasets

- **VL-RewardBench** — total 1250; splits: test (1250)

## Metrics

- `Overall Accuracy` **(primary)** — range: percent
  - Percentage of model decisions that match the ground-truth human preferences across all test samples.
- `Macro Average Accuracy` — range: percent
  - Mean accuracy computed independently across each task category, then averaged to mitigate task distribution imbalance.

## Input / output format

**Input**: A multimodal query (image + text) paired with two candidate responses (one preferred, one rejected) formatted via a standardized evaluation template.

**Output**: A binary preference decision indicating which of the two candidate responses the model selects as preferred.

## Scoring recipe

```python
def evaluate(model, test_set, K=5):
    correct = 0
    for sample in test_set:
        votes = []
        for _ in range(K):
            order = random.shuffle([sample.resp_A, sample.resp_B])
            pred = model.generate(order)
            votes.append(pred)
        final_pred = majority_vote(votes)
        if final_pred == sample.human_preference:
            correct += 1
    return correct / len(test_set)
```

## Common pitfalls

- Positional bias: models may favor responses based on presentation order if not randomized and aggregated via majority voting.
- Task distribution imbalance: overall accuracy can be skewed by easier categories; macro average accuracy is required for fair cross-category comparison.
- Decoding sensitivity: results vary if temperature/top-p are not fixed (paper uses 0.2/0.2).

## Evidence (verbatim from paper)

> We calculate two primary metrics: Overall Accuracy: percentage of model decisions matching human preferences, and Macro Average Accuracy: mean accuracy across different task categories, addressing the task distribution imbalance. To mitigate positional bias, where models favor responses based on their presentation order, we conduct K independent evaluations for each preference pair with randomized response ordering. The final preference is determined through majority voting across these K runs.

## Citation

```bibtex
@misc{li2024vlrewardbench,
  title={VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models},
  author={Lei Li et al. (2024)},
  year={2024},
  note={arXiv:2411.17451}
}
```

- arXiv: 2411.17451