Prometheus 2 Eval

---
name: prometheus-2-eval
description: This evaluation protocol assesses the capability of language models to act as automated judges for other language models. It probes two distinct paradigms: direct assessment, where a model scores a single response against a reference or rubric, and pairwise ranking, where a model selects the preferred response between two candidates. The benchmarks cover instruction-following, alignment, and fine-grained custom criteria. Use when the user wants to benchmark on Vicuna Bench, MT Bench, FLASK, Feedback Bench, HHH Alignment, MT Bench Human Judgment, Auto-J Eval, Preference Bench, or asks about evaluating this task. Reports Pearson correlation, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.01535
  bibtex_key: kim2024prometheus2
  confidence: high
---

# prometheus-2-eval

> Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models — Seungone Kim et al. (2024) (arXiv:2405.01535, 2024)

## What this evaluates

This evaluation protocol assesses the capability of language models to act as automated judges for other language models. It probes two distinct paradigms: direct assessment, where a model scores a single response against a reference or rubric, and pairwise ranking, where a model selects the preferred response between two candidates. The benchmarks cover instruction-following, alignment, and fine-grained custom criteria.

## Datasets

- **Vicuna Bench** — total 80; splits: test (80)
- **MT Bench** — total 80; splits: test (80)
- **FLASK** — total 200; splits: test (200)
- **Feedback Bench** — total 200; splits: test (200)
- **HHH Alignment** — total 221; splits: test (221)
- **MT Bench Human Judgment** — total 80; splits: test (80)
- **Auto-J Eval** — total 58; splits: test (58)
- **Preference Bench** — total 200; splits: test (200)

## Metrics

- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Measures the linear correlation coefficient between the evaluator model's predicted scores and the reference human/proprietary scores. Calculated as the covariance of the two variables divided by the product of their standard deviations.
- `Spearman correlation` — range: [-1, 1]
  - Measures the monotonic relationship between predicted and reference scores based on their rank order. Computed as the Pearson correlation of the rank-transformed variables.
- `Kendall-Tau correlation` — range: [-1, 1]
  - Measures the ordinal association between predicted and reference scores by counting concordant and discordant pairs. Defined as (concordant - discordant) / total pairs.
- `Accuracy` **(primary)** — range: [0, 1]
  - The proportion of correct predictions where the evaluator model's ranking choice matches the human judgment. Calculated as the number of matching predictions divided by the total number of evaluated pairs.

## Input / output format

**Input**: Direct assessment: A prompt, a candidate response, and an appended reference answer or evaluation rubric. Pairwise ranking: A prompt and two candidate responses (Response A and Response B) provided without a reference answer.

**Output**: Direct assessment: A numerical score reflecting response quality. Pairwise ranking: A categorical label indicating 'win', 'lose', or 'tie' for the preferred response.

## Scoring recipe

```python
def evaluate_direct_assessment(pred_scores, ref_scores):
    pearson = pearsonr(pred_scores, ref_scores)
    spearman = spearmanr(pred_scores, ref_scores)
    kendall = kendalltau(pred_scores, ref_scores)
    return pearson, spearman, kendall

def evaluate_pairwise_ranking(pred_labels, ref_labels):
    valid_mask = [r != 'tie' for r in ref_labels]
    pred = [p for p, m in zip(pred_labels, valid_mask) if m]
    ref = [r for r, m in zip(ref_labels, valid_mask) if m]
    correct = sum(p == r for p, r in zip(pred, ref))
    return correct / len(ref) if ref else 0.0
```

## Common pitfalls

- Confusing reference-based direct assessment with reference-free evaluation; the primary protocol explicitly appends the reference answer as input.
- Handling 'tie' responses inconsistently; the paper evaluates both by excluding ties ('w/o tie') and by grouping ties into direct assessment scores ('w/ tie').
- Assuming uniform scoring scales across benchmarks; FLASK and Feedback Bench use custom rubrics, while others rely on general helpfulness/harmlessness criteria.

## Evidence (verbatim from paper)

> In direct assessment, we conduct reference-based evaluations by appending the reference answer as the input. We use Pearson, Spearman, and Kendall-Tau as performance metrics to measure scoring correlations against reference evaluators. Moreover, we include the results of the reference-free direct assessment evaluation in Appendix[F]. In pairwise ranking, we conduct reference-free evaluations. Based on judgments assigned by humans, we use accuracy as our metric to measure agreement between evaluator LMs and humans.

## Citation

```bibtex
@misc{kim2024prometheus2,
  title={Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models},
  author={Seungone Kim et al. (2024)},
  year={2024},
  note={arXiv:2405.01535}
}
```

- arXiv: 2405.01535