Mixeval Eval

---
name: mixeval-eval
description: Evaluates LLMs on a dynamically mixed benchmark of real-world web-mined queries and existing datasets to measure alignment with human preferences. It probes a model's general capability, reasoning, and instruction-following across diverse domains, correlating performance with Chatbot Arena Elo scores. Use when the user wants to benchmark on MixEval, MixEval-Hard, or asks about evaluating this task. Reports Spearman's ranking correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.06565
  bibtex_key: ni2024mixeval
  confidence: high
---

# mixeval-eval

> MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures — Ni et al. (2024) (arXiv:2406.06565, 2024)

## What this evaluates

Evaluates LLMs on a dynamically mixed benchmark of real-world web-mined queries and existing datasets to measure alignment with human preferences. It probes a model's general capability, reasoning, and instruction-following across diverse domains, correlating performance with Chatbot Arena Elo scores.

## Datasets

- **MixEval** — total ?; splits: test (-1)
- **MixEval-Hard** — total ?; splits: test (-1)

## Metrics

- `Spearman's ranking correlation` **(primary)** — range: [-1, 1]
  - Spearman's rank correlation coefficient measuring the monotonic relationship between a model's score on the benchmark and its Chatbot Arena Elo rating. Computed as the rank correlation between the two score vectors.

## Input / output format

**Input**: Prompted with benchmark questions using official chat templates or FastChat templates. Base models receive a 5-shot in-context learning setup with examples.

**Output**: Free-form text responses or multiple-choice selections. Responses are parsed by GPT-3.5-Turbo-0125 to extract scores or choices.

## Scoring recipe

```python
predictions = []
for question in benchmark:
    response = model.generate(question)
    if question.type == "free-form":
        score = gpt35_parser.score(response)
    else:
        choice = gpt35_parser.extract_choice(response)
        score = 1 if choice == question.gold else 0
    predictions.append(score)
arena_elo_scores = get_arena_elo(model_name)
correlation = spearmanr(predictions, arena_elo_scores)
return correlation
```

## Common pitfalls

- Using rule-based parsing instead of LLM-based parsing (GPT-3.5-Turbo-0125) leads to significant inaccuracies in scoring free-form or multiple-choice answers.
- Assuming the benchmark correlation with Arena Elo provides an exact equivalence; the paper notes the estimation remains approximate due to outliers.
- Evaluating base models without the specified 5-shot setting or chat models without official/FastChat templates, which breaks consistency with the reported results.

## Evidence (verbatim from paper)

> ρ and e denote the Spearman's ranking correlation and the root mean square error of the linear fit respectively.

## Citation

```bibtex
@misc{ni2024mixeval,
  title={MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures},
  author={Ni et al. (2024)},
  year={2024},
  note={arXiv:2406.06565}
}
```

- arXiv: 2406.06565