Kimi K1.5 Benchmark Eval

---
name: kimi-k1.5-benchmark-eval
description: Evaluates multimodal reasoning, coding, and instruction-following capabilities across text, code, and vision tasks using a standardized suite of academic benchmarks. Use when the user wants to benchmark on MMLU, IF-Eval, CLUEWSC, C-EVAL, HumanEval-Mul, LiveCodeBench, Codeforces, AIME 2024, MATH-500, MMMU, MATH-Vision, MathVista, or asks about evaluating this task. Reports exact-match accuracy (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.12599
  bibtex_key: kimi2025k1.5
  confidence: high
---

# kimi-k1.5-benchmark-eval

> Kimi k1.5: Scaling Reinforcement Learning with LLMs — Kimi Team (2025) (arXiv:2501.12599, 2025)

## What this evaluates

Evaluates multimodal reasoning, coding, and instruction-following capabilities across text, code, and vision tasks using a standardized suite of academic benchmarks.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **IF-Eval** — total ?; splits: test (-1)
- **CLUEWSC** — total ?; splits: test (-1)
- **C-EVAL** — total ?; splits: test (-1)
- **HumanEval-Mul** — total ?; splits: test (-1)
- **LiveCodeBench** — total ?; splits: test (-1)
- **Codeforces** — total ?; splits: test (-1)
- **AIME 2024** — total ?; splits: test (-1)
- **MATH-500** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: val (-1)
- **MATH-Vision** — total ?; splits: full (-1)
- **MathVista** — total ?; splits: test (-1)

## Metrics

- `exact-match accuracy (EM)` **(primary)** — range: [0, 1]
  - 1 if the model's generated answer exactly matches the ground truth answer, 0 otherwise. Averaged over all instances.
- `pass@1` — range: [0, 1]
  - 1 if the single generated solution passes the provided test cases or matches the expected output, 0 otherwise. Averaged over all instances.
- `prompt strict accuracy` — range: [0, 1]
  - Exact match score under strict prompt-following constraints as defined by the IF-Eval benchmark.
- `percentile` — range: [0, 100]
  - Ranking percentile of the model's score relative to a reference distribution of model scores on the benchmark.

## Input / output format

**Input**: Standard benchmark prompts for text, code, and vision-language tasks. Vision tasks include image inputs paired with text questions.

**Output**: Model-generated text or code responses. For code benchmarks, executable code snippets are generated.

## Scoring recipe

```python
def score_em(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip()) / len(gold)

def score_pass1(predictions, gold_tests):
    passed = 0
    for pred, tests in zip(predictions, gold_tests):
        if run_and_pass(pred, tests):
            passed += 1
    return passed / len(predictions)
```

## Common pitfalls

- Detailed evaluation setup (temperature, max tokens, sampling strategy) is deferred to Appendix C and not specified in the main text.
- Vision-language model results for baselines are sourced from the OpenCompass platform rather than directly evaluated by the authors.
- Metrics vary significantly across benchmarks (EM, Pass@1, Percentile, Prompt Strict), requiring careful alignment when comparing scores.

## Evidence (verbatim from paper)

> Our benchmarks primarily consist of the following three categories: Text Benchmark: MMLU*, IF-Eval*, CLUEWSC*, C-EVAL* ... Reasoning Benchmark: HumanEval-Mul, LiveCodeBench*, Codeforces, AIME 2024, MATH-500* ... Vision Benchmark: MMMU*, MATH-Vision*, MathVista* ... Table 2: Performance of Kimi k1.5 long-CoT and flagship open-source and proprietary models. (Metric) ... MATH-500 (EM) ... AIME 2024 (Pass@1) ... LiveCodeBench (Pass@1)

## Citation

```bibtex
@misc{kimi2025k1.5,
  title={Kimi k1.5: Scaling Reinforcement Learning with LLMs},
  author={Kimi Team (2025)},
  year={2025},
  note={arXiv:2501.12599}
}
```

- arXiv: 2501.12599