Ultrachat Eval

---
name: ultrachat-eval
description: Evaluates a chat model's ability to generate accurate, informative, and correct responses across diverse domains including commonsense, world knowledge, professional knowledge, mathematics, reasoning, and writing. It probes both factual correctness and response quality using automated LLM-based pairwise and independent scoring. Use when the user wants to benchmark on UltraChat Evaluation Set, or asks about evaluating this task. Reports ChatGPT scoring.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14233
  bibtex_key: ding2023ultrachat
  confidence: high
---

# ultrachat-eval

> Enhancing Chat Language Models by Scaling High-quality Instructional Conversations — Ding et al. (2023) (arXiv:2305.14233, 2023)

## What this evaluates

Evaluates a chat model's ability to generate accurate, informative, and correct responses across diverse domains including commonsense, world knowledge, professional knowledge, mathematics, reasoning, and writing. It probes both factual correctness and response quality using automated LLM-based pairwise and independent scoring.

## Datasets

- **UltraChat Evaluation Set** — total ?; splits: test (-1); repo https://github.com/thunlp/UltraChat

## Metrics

- `ChatGPT scoring` **(primary)** — range: [1, 10]
  - Average score from 1 to 10 assigned by ChatGPT to each response, with the evaluation prompt explicitly prioritizing correctness over other factors like informativeness.
- `Win/Tie/Lose rate` — range: percent
  - Percentage of pairwise comparisons where a model's response scores higher (Win), equal (Tie), or lower (Lose) than a baseline's response on the same question.
- `TruthfulQA accuracy` — range: [0, 1]
  - Accuracy of the model's true/false judgment on multiple-choice answer candidates from the TruthfulQA benchmark.

## Input / output format

**Input**: A question or instruction, optionally paired with a second model response for pairwise comparison, and a system prompt if used.

**Output**: A score from 1 to 10 with reasoning (for ChatGPT evaluation), or a True/False judgment (for TruthfulQA).

## Scoring recipe

```python
def evaluate_pairwise(question, resp_a, resp_b):
    prompt = f'Q: {question}\nA: {resp_a}\nB: {resp_b}\nScore 1-10, prioritize correctness.'
    scores = call_chatgpt(prompt) # returns {'A': int, 'B': int}
    return scores['A'], scores['B']

def calc_win_rate(results):
    wins = sum(1 for a, b in results if a > b)
    ties = sum(1 for a, b in results if a == b)
    loses = sum(1 for a, b in results if a < b)
    return wins, ties, loses

# For independent scoring, average ChatGPT scores across all questions.
```

## Common pitfalls

- Response presentation order significantly biases ChatGPT scores; the protocol requires randomizing the order of responses for each question.
- Pairwise comparison is noted as unstable by the authors, so independent scoring must be used alongside pairwise results for reliable assessment.
- TruthfulQA is evaluated via true/false judgment on multiple-choice candidates, not standard multiple-choice selection accuracy.

## Evidence (verbatim from paper)

> We use ChatGPT to compare our model output with each baseline model on each question. Specifically, we input the question and a pair of independent answers from two models respectively, and task ChatGPT with scoring each response on a scale of 1 to 10 and providing reasoning for the given score. Our evaluation prompt is designed to prioritize correctness over other factors such as informativeness. Additionally, we discover that the order in which the responses are presented significantly affects the evaluation results. To address this issue, we randomly determine the order of the responses for each question.

## Citation

```bibtex
@misc{ding2023ultrachat,
  title={Enhancing Chat Language Models by Scaling High-quality Instructional Conversations},
  author={Ding et al. (2023)},
  year={2023},
  note={arXiv:2305.14233}
}
```

- arXiv: 2305.14233