Phi 3 Academic Benchmarks Eval

---
name: phi-3-academic-benchmarks-eval
description: Evaluates language models on reasoning, common sense, logical reasoning, knowledge retrieval, and coding capabilities across a diverse suite of standard academic benchmarks. Use when the user wants to benchmark on MMLU, HellaSwag, ANLI, GSM-8K, MATH, MedQA, AGIEval, TriviaQA, Arc-C, Arc-E, PIQA, SociQA, BigBench-Hard, WinoGrande, OpenBookQA, BoolQ, CommonSenseQA, TruthfulQA, HumanEval, MBPP, GPQA, MT Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.14219
  bibtex_key: abdin2024phi3
  confidence: high
---

# phi-3-academic-benchmarks-eval

> Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone — Abdin et al. (2024) (arXiv:2404.14219, 2024)

## What this evaluates

Evaluates language models on reasoning, common sense, logical reasoning, knowledge retrieval, and coding capabilities across a diverse suite of standard academic benchmarks.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **ANLI** — total ?; splits: test (-1)
- **GSM-8K** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **AGIEval** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **Arc-C** — total ?; splits: test (-1)
- **Arc-E** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **SociQA** — total ?; splits: test (-1)
- **BigBench-Hard** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **OpenBookQA** — total ?; splits: test (-1)
- **BoolQ** — total ?; splits: test (-1)
- **CommonSenseQA** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)
- **MBPP** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **MT Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of test instances. For multiple-choice benchmarks, the model's selected option is compared to the gold label. For code benchmarks (HumanEval, MBPP), it is the pass rate of generated code against unit tests.

## Input / output format

**Input**: A few-shot prompt template containing k examples (k varies by benchmark: 0, 2, 3, 5, 7, 8, or 10 shots) followed by the target question or task. Chain-of-thought (CoT) prompting is explicitly enabled for GSM-8K, MATH, BigBench-Hard, and GPQA.

**Output**: Model-generated text response. For multiple-choice benchmarks, the predicted option letter or text. For code benchmarks, a Python function. For MT Bench, a conversational response scored by an LLM judge.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for p, g in zip(predictions, golds):
        if normalize_answer(p) == normalize_answer(g):
            correct += 1
    return (correct / len(golds)) * 100

# For MT-Bench:
# score = llm_judge_score(prediction, reference, criteria)
# return average score over 2 rounds
```

## Common pitfalls

- Prompts use a proprietary Microsoft internal template rather than standard open-source templates, which can cause score variations compared to other evaluations.
- Chain-of-thought (CoT) is only enabled for specific benchmarks (GSM-8K, MATH, BigBench-Hard, GPQA), not universally across all tasks.
- Temperature is strictly fixed at 0 for all evaluations, which may differ from standard zero-shot or higher-temperature settings used elsewhere.

## Evidence (verbatim from paper)

> As is now standard, we use few-shot prompts to evaluate the models, at temperature 0. The prompts and number of shots are part of a Microsoft internal tool to evaluate language models, and in particular we did no optimization to the pipeline for the phi-3 models. All the reported numbers are produced with the exact same pipeline to ensure that the numbers are comparable.

## Citation

```bibtex
@misc{abdin2024phi3,
  title={Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author={Abdin et al. (2024)},
  year={2024},
  note={arXiv:2404.14219}
}
```

- arXiv: 2404.14219