Ming Moe Medical Eval

---
name: ming-moe-medical-eval
description: Evaluates large language models on a comprehensive suite of medical natural language processing tasks and medical licensing examinations. It probes the model's ability to process clinical text, perform information extraction, and demonstrate domain-specific knowledge and reasoning for medical exams. Use when the user wants to benchmark on CBLUE (via PromptCBLUE), MedQA, MedMCQA, CMB, CMExam, MMLU (medical subset), C-Eval (medical subset), CMMLU (medical subset), 2023 Chinese National Pharmacist Licensure Examination, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.09027
  bibtex_key: liao2024mingmoe
  confidence: high
---

# ming-moe-medical-eval

> MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts — Liao et al. (2024) (arXiv:2404.09027, 2024)

## What this evaluates

Evaluates large language models on a comprehensive suite of medical natural language processing tasks and medical licensing examinations. It probes the model's ability to process clinical text, perform information extraction, and demonstrate domain-specific knowledge and reasoning for medical exams.

## Datasets

- **CBLUE (via PromptCBLUE)** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: validation (-1)
- **MedMCQA** — total ?; splits: validation (-1)
- **CMB** — total ?; splits: validation (-1)
- **CMExam** — total ?; splits: test (-1)
- **MMLU (medical subset)** — total ?; splits: test (-1)
- **C-Eval (medical subset)** — total ?; splits: test (-1)
- **CMMLU (medical subset)** — total ?; splits: test (-1)
- **2023 Chinese National Pharmacist Licensure Examination** — total ?; splits: test (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Percentage of correctly predicted options or task-specific scores (e.g., F1, Accuracy) averaged across tasks. For multiple-choice exams, exact match between the model's selected option and the ground truth.

## Input / output format

**Input**: Multiple-choice medical questions or clinical text prompts formatted for LLM consumption (e.g., via PromptCBLUE for CBLUE tasks).

**Output**: Model generates a textual response, typically selecting a letter option (A, B, C, D) or providing a direct answer. For CBLUE tasks, task-specific structured outputs.

## Scoring recipe

```python
def compute_score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_opt = normalize_option(pred)
        if pred_opt == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- MedQA, MedMCQA, and CMB are evaluated on their validation sets according to the paper, not the standard test sets.
- For general benchmarks (MMLU, C-Eval, CMMLU), only medical-domain questions are extracted and evaluated, not the full benchmark.
- CBLUE tasks use PromptCBLUE to convert structured tasks into pure text prompts, which may affect baseline comparability if not applied consistently.

## Evidence (verbatim from paper)

> The examination benchmarks include: the US test set of MedQA(Jin et al., 2021), the development set of MedMCQA(Pal et al., 2022), and two comprehensive Chinese medical exam datasets, CMB(Wang et al., 2023c) and CMExam(Liu et al., 2024a). We also collect the medical parts of the general benchmarks, which include: MMLU(Hendrycks et al., 2020), C-Eval(Huang et al., 2023), and CMMLU(Li et al., 2023a). Evaluation was done using validation data for MedQA, MedMCQA, and CMB.

## Citation

```bibtex
@misc{liao2024mingmoe,
  title={MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts},
  author={Liao et al. (2024)},
  year={2024},
  note={arXiv:2404.09027}
}
```

- arXiv: 2404.09027