Open Llm Leaderboard Eval

---
name: open-llm-leaderboard-eval
description: This benchmark evaluates large language models on their ability to answer open-style questions across diverse knowledge and reasoning domains. It specifically probes whether models rely on selection bias and random guessing in multiple-choice formats versus genuinely understanding and generating correct open-ended responses. Use when the user wants to benchmark on MMLU, ARC, MedMCQA, PIQA, CommonsenseQA, OpenBookQA, RACE, WinoGrande, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.07545
  bibtex_key: myrzakhan2024openllmleaderboard
  confidence: high
---

# open-llm-leaderboard-eval

> Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena — Myrzakhan et al. (2024) (arXiv:2406.07545, 2024)

## What this evaluates

This benchmark evaluates large language models on their ability to answer open-style questions across diverse knowledge and reasoning domains. It specifically probes whether models rely on selection bias and random guessing in multiple-choice formats versus genuinely understanding and generating correct open-ended responses.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **ARC** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **CommonsenseQA** — total ?; splits: test (-1)
- **OpenBookQA** — total ?; splits: test (-1)
- **RACE** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For MCQ, it is the proportion of responses matching the ground truth ID. For OSQ, it is the proportion of responses judged correct by a GPT-4-0125-preview judge against the ground truth answer.

## Input / output format

**Input**: Dataset questions formatted as either multiple-choice (MCQ) with options or open-style questions (OSQ) requiring free-text generation. Prompts are provided in Appendix D.

**Output**: For MCQ: a single choice ID or letter. For OSQ: a free-text answer string.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, mode='osq'):
    correct = 0
    for pred, g in zip(predictions, gold):
        if mode == 'mcq':
            if pred == g: correct += 1
        else:
            judge_out = llm_judge(prompt=f'Q: {g} A: {pred}')
            if judge_out == 'correct': correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- HellaSwag contains multiple plausible continuations, making single-ground-truth evaluation unreliable; the authors explicitly omit it from the final leaderboard.
- Open-style questions naturally yield lower accuracy (~25% drop) than MCQ due to the absence of selection bias, which should not be conflated with model incompetence.
- LLM-as-a-judge evaluation for OSQ depends heavily on the specific GPT-4 prompt and may still struggle with semantically equivalent but differently phrased correct answers.

## Evidence (verbatim from paper)

> The results in Table [4] and Figure [4] are based on filtered questions. They show that every model experiences a significant drop in the accuracy for OSQ compared to MCQ. On average, the accuracy of OSQ is lower than MCQ by about 25% for all models. This result can correlate with our concern that the model will “randomly guess” to correct choices but it cannot answer.

## Citation

```bibtex
@misc{myrzakhan2024openllmleaderboard,
  title={Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena},
  author={Myrzakhan et al. (2024)},
  year={2024},
  note={arXiv:2406.07545}
}
```

- arXiv: 2406.07545