Mm Alignbench Eval

---
name: mm-alignbench-eval
description: Evaluates how well multi-modal large language models align with human preferences when answering open-ended questions about diverse images. It probes the model's ability to follow complex instructions, handle real-world scenarios, and produce responses that match human expectations better than baseline models. Use when the user wants to benchmark on MM-AlignBench, or asks about evaluating this task. Reports Win Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.18411
  bibtex_key: [REDACTED]
  confidence: high
---

# mm-alignbench-eval

> OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference — Zhao et al. (2025) (arXiv:2502.18411, 2025)

## What this evaluates

Evaluates how well multi-modal large language models align with human preferences when answering open-ended questions about diverse images. It probes the model's ability to follow complex instructions, handle real-world scenarios, and produce responses that match human expectations better than baseline models.

## Datasets

- **MM-AlignBench** — total ?; splits: test (-1); repo https://github.com/PhoenixZ810/OmniAlign-V

## Metrics

- `Win Rate` **(primary)** — range: percent
  - Percentage of pairwise comparisons where the evaluated model's response is preferred over a reference model's response. Calculated as (Count(B) + Count(B+)) / Total Comparisons * 100.
- `Reward` — range: other
  - Average preference score or margin derived from pairwise comparisons, indicating the strength of alignment relative to the reference model.

## Input / output format

**Input**: A single image paired with an open-ended question or instruction prompt.

**Output**: A free-form text response generated by the MLLM.

## Scoring recipe

```python
# Pairwise evaluation protocol
responses = model.generate(image, prompt)
reference_responses = get_reference_responses(image, prompt) # e.g., Claude-3-V-Sonnet
comparisons = pairwise_compare(responses, reference_responses) # Human or judge model
win_rate = (count(comparisons == 'B') + count(comparisons == 'B+')) / len(comparisons) * 100
reward = average_preference_score(comparisons)
return win_rate, reward
```

## Common pitfalls

- The benchmark uses pairwise comparisons against a strong reference model (e.g., Claude-3-V-Sonnet), so scores are relative, not absolute.
- Win Rate and Reward are reported together; confusing the raw count of B+/B/T/W/W+ with the final Win Rate percentage is a common error.
- The evaluation focuses on open-ended, real-world questions rather than standard VQA, so models optimized for closed-set VQA may underperform despite high factual accuracy.

## Evidence (verbatim from paper)

> After applying DPO with OmniAlign-V-DPO, LLaVA-OA-32B-DPO achieves winning rate of 72.6 with an average reward of +33.5, surpassing the performance of Qwen2VL-72B.

## Citation

```bibtex
@misc{zhao2025omnialignv,
  title={OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2502.18411}
}
```

- arXiv: 2502.18411