Kokushimd 10 Eval

---
name: kokushimd-10-eval
description: This benchmark evaluates large language models' ability to reason through Japanese national healthcare licensing examinations across ten medical professions. It probes domain-specific clinical knowledge, multimodal image interpretation, and high-stakes decision-making under strict, profession-specific passing criteria. Use when the user wants to benchmark on KokushiMD-10, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.11114
  bibtex_key: [REDACTED]
  confidence: high
---

# kokushimd-10-eval

> KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations — Liu et al. (2025) (arXiv:2506.11114, 2025)

## What this evaluates

This benchmark evaluates large language models' ability to reason through Japanese national healthcare licensing examinations across ten medical professions. It probes domain-specific clinical knowledge, multimodal image interpretation, and high-stakes decision-making under strict, profession-specific passing criteria.

## Datasets

- **KokushiMD-10** — total ?; splits: test (-1); repo https://github.com/juniorliu95/KokushiMD-10

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions. Multiple-choice questions require exact set match of selected options; partial credit is not awarded.
- `official_pass_fail` — range: other
  - Binary outcome: 1 if the model's exam score meets the official profession-specific threshold, 0 otherwise.

## Input / output format

**Input**: Japanese system prompt specifying role and output format, followed by a user prompt containing the question text and, for multimodal runs, associated clinical images.

**Output**: Strictly formatted answer as defined in the prompt (e.g., exact option letters for multiple-choice, numerical value, or text fill-in).

## Scoring recipe

```python
def evaluate_exam(predictions, gold_answers, question_types, official_threshold):
    correct = 0
    total = len(gold_answers)
    for pred, gold, qtype in zip(predictions, gold_answers, question_types):
        if qtype == 'Multiple Choice':
            if set(pred) == set(gold): correct += 1
        else:
            if pred == gold: correct += 1
    accuracy = correct / total
    passed = accuracy >= official_threshold
    return accuracy, passed
```

## Common pitfalls

- Multiple-choice questions use strict exact-match scoring; selecting a subset of correct options yields zero points.
- Some exams contain 'forbidden options' that trigger automatic failure if selected, overriding other correct answers.
- Passing thresholds and question weights are highly variable across professions and years, requiring profession-specific evaluation scripts.

## Evidence (verbatim from paper)

> Questions are categorized into four types: Single Answer, Multiple Choice, Numerical Calculation, and Blank. Each type is scored with strict correctness rules—for example, multiple-choice questions are marked incorrect unless the selected options exactly match the correct set. Scenario-based questions in nursing and midwifery are evaluated with higher weight to reflect real-world decision-making. We report per-type accuracy, as well as domain-specific thresholds aligned with official exam standards. A model is considered to have passed an exam if its score meets the official passing threshold for that exam.

## Citation

```bibtex
@misc{liu2025kokushimd10,
  title={KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.11114}
}
```

- arXiv: 2506.11114