Medical Llm Merging Eval

---
name: medical-llm-merging-eval
description: Evaluates the effectiveness of various model merging techniques for consolidating knowledge in medical large language models. It probes whether merged models can outperform their base and parent models across diverse medical and general reasoning benchmarks. Use when the user wants to benchmark on MedQA, PubMedQA, HellaSwag, MedMCQA, MMLU Professional Medicine, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.13373
  bibtex_key: timilsina2025hierarchical
  confidence: high
---

# medical-llm-merging-eval

> A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs — Timilsina et al. (2025) (arXiv:2511.13373, 2025)

## What this evaluates

Evaluates the effectiveness of various model merging techniques for consolidating knowledge in medical large language models. It probes whether merged models can outperform their base and parent models across diverse medical and general reasoning benchmarks.

## Datasets

- **MedQA** — total ?; splits: test (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **MMLU Professional Medicine** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly answered multiple-choice questions out of the total number of questions.
- `standard error` — range: [0, 1]
  - The standard deviation of the accuracy scores across runs or folds, used to report confidence intervals.

## Input / output format

**Input**: Multiple-choice questions from medical and general reasoning benchmarks provided to merged LLMs.

**Output**: Model-generated answer choices, evaluated for correctness against gold labels.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Default pruning density for DARE-TIES (0.6) may destroy valuable medical knowledge; higher densities (0.8-0.95) are often needed.
- Merged models do not automatically outperform the base model on all tasks; broad knowledge benchmarks like MMLU Professional Medicine may see no improvement.

## Evidence (verbatim from paper)

> The evaluation across five medical and general reasoning benchmarks revealed significant performance differences among the merging techniques. The detailed results, including accuracy and standard error for each model, are presented in Table[IV].

## Citation

```bibtex
@misc{timilsina2025hierarchical,
  title={A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs},
  author={Timilsina et al. (2025)},
  year={2025},
  note={arXiv:2511.13373}
}
```

- arXiv: 2511.13373