Evaluator Lm Eval

---
name: evaluator-lm-eval
description: This protocol evaluates an LLM's ability to act as a fine-grained text evaluator using custom score rubrics. It tests both absolute grading (assigning a 1–5 score and generating feedback based on a rubric and reference answer) and ranking grading (predicting human preference between two responses). Use when the user wants to benchmark on Feedback Bench, Vicuna Bench, MT Bench, FLASK Eval, MT Bench Human Judgments, HHH Alignment, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.08491
  bibtex_key: kim2023prometheus
  confidence: high
---

# evaluator-lm-eval

> Prometheus: Inducing Fine-grained Evaluation Capability in Language Models — Seungone Kim et al. (2023) (arXiv:2310.08491, 2023)

## What this evaluates

This protocol evaluates an LLM's ability to act as a fine-grained text evaluator using custom score rubrics. It tests both absolute grading (assigning a 1–5 score and generating feedback based on a rubric and reference answer) and ranking grading (predicting human preference between two responses).

## Datasets

- **Feedback Bench** — total 2000; splits: test (2000); repo https://github.com/kaistAI/Prometheus
- **Vicuna Bench** — total 80; splits: test (80)
- **MT Bench** — total 80; splits: test (80)
- **FLASK Eval** — total 200; splits: test (200)
- **MT Bench Human Judgments** — total ?; splits: test (-1)
- **HHH Alignment** — total 221; splits: test (221)

## Metrics

- `Pearson correlation` **(primary)** — range: [−1, 1]
  - Measures the linear correlation between the evaluator model's scores and human/GPT-4 reference scores across all instances.
- `Kendall-Tau correlation` — range: [−1, 1]
  - Measures the ordinal association between predicted and reference scores by counting concordant and discordant pairs.
- `Spearman correlation` — range: [−1, 1]
  - A rank-based measure of statistical dependence between the evaluator's scores and reference scores.
- `accuracy` — range: [0, 1]
  - The fraction of response pairs where the evaluator correctly predicts the human-preferred response.

## Input / output format

**Input**: Instruction, response(s) to evaluate, custom score rubric, and reference answer(s). For ranking tasks, two candidate responses are provided alongside the instruction and rubric.

**Output**: A textual feedback critique and a numerical score on a 1–5 scale. For ranking evaluation, the model scores each candidate independently.

## Scoring recipe

```python
def score_absolute(model_scores, ref_scores):
    return pearsonr(model_scores, ref_scores)

def score_ranking(model, pairs):
    correct = 0
    for pair in pairs:
        s1 = model.score(pair.r1, temp=1.0)
        s2 = model.score(pair.r2, temp=1.0)
        while s1 == s2:
            s1 = model.score(pair.r1, temp=1.0)
            s2 = model.score(pair.r2, temp=1.0)
        if (s1 > s2) == (pair.human_pref == 1):
            correct += 1
    return correct / len(pairs)
```

## Common pitfalls

- Ranking evaluation uses temperature 1.0 and iterative independent scoring to break ties, which the authors explicitly note is not a fair comparison to standard paired ranking models.
- Reference answers for several benchmarks (Vicuna, MT Bench) are synthetically generated by GPT-4 rather than human-written, which may bias the evaluation.
- Absolute grading requires the model to score without an opponent, making it inherently harder than ranking but more practical for real-world use.

## Evidence (verbatim from paper)

> For the experiments that measure the correlation, we use 3 different correlation metrics: Pearson, Kdendall-Tau, and Spearman. For measuring the quality of the generated feedback, we conduct a pairwise comparison between the feedback generated by Prometheus, GPT-3.5-Turbo, and GPT-4, asking human evaluators to choose which has better quality and why they thought so.

## Citation

```bibtex
@misc{kim2023prometheus,
  title={Prometheus: Inducing Fine-grained Evaluation Capability in Language Models},
  author={Seungone Kim et al. (2023)},
  year={2023},
  note={arXiv:2310.08491}
}
```

- arXiv: 2310.08491