Mt Bench Eval

---
name: mt-bench-eval
description: Evaluates the conversational quality and instruction-following capability of aligned language models across multiple knowledge domains. It also measures whether alignment fine-tuning causes regression in base reasoning, truthfulness, and commonsense capabilities. Use when the user wants to benchmark on MT-Bench, Open LLM Leaderboard Benchmarks, or asks about evaluating this task. Reports MT-Bench Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.14723
  bibtex_key: saeidi2024insights
  confidence: high
---

# mt-bench-eval

> Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks — Saeidi et al. (2024) (arXiv:2404.14723, 2024)

## What this evaluates

Evaluates the conversational quality and instruction-following capability of aligned language models across multiple knowledge domains. It also measures whether alignment fine-tuning causes regression in base reasoning, truthfulness, and commonsense capabilities.

## Datasets

- **MT-Bench** — total 160; splits: test (160)
- **Open LLM Leaderboard Benchmarks** — total ?; splits: test (-1)

## Metrics

- `MT-Bench Average Score` **(primary)** — range: [0, 10]
  - Mean of GPT-4 ratings (1-10) for the first turn response and the predefined follow-up question response.
- `Accuracy` — range: percent
  - Percentage of correctly predicted answers across ARC, HellaSwag, MMLU, TruthfulQA, GSM8k, and Winogrande.

## Input / output format

**Input**: MT-Bench: a user prompt followed by a predefined follow-up question. Open LLM Benchmarks: standard dataset-specific prompts (multiple-choice or open-ended).

**Output**: MT-Bench: a text response to the prompt/follow-up. Open LLM Benchmarks: a predicted answer or option.

## Scoring recipe

```python
# MT-Bench
mt_bench_score = (gpt4_rate(response_turn1) + gpt4_rate(response_turn2)) / 2.0

# Open LLM Benchmarks
accuracy = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold) / len(predictions)
```

## Common pitfalls

- MT-Bench relies entirely on GPT-4 for scoring, which may introduce bias or inconsistency compared to human evaluation.
- Open LLM benchmarks measure accuracy on static test sets, which may not reflect real-world conversational quality or alignment improvements.
- The paper evaluates both SFT+alignment and SFT-free alignment, but results are reported separately, requiring careful comparison of base models.

## Evidence (verbatim from paper)

> We employ MT-Bench *zheng2023judging*, which comprises 160 questions spanning eight knowledge domains for evaluation. In MT-Bench, models are tasked with answering a question and subsequently addressing a predefined follow-up question. Model responses are rated on a scale of 1 to 10 by GPT-4, and the final score is calculated as the mean over the two turns.

## Citation

```bibtex
@misc{saeidi2024insights,
  title={Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks},
  author={Saeidi et al. (2024)},
  year={2024},
  note={arXiv:2404.14723}
}
```

- arXiv: 2404.14723