Principlismqa Eval

---
name: principlismqa-eval
description: Evaluates large language models' ability to reason about medical ethics using the Principlism framework (autonomy, non-maleficence, beneficence, justice). It probes both theoretical knowledge of ethical principles and their practical application to complex, open-ended clinical dilemmas. Use when the user wants to benchmark on PrinciplismQA, or asks about evaluating this task. Reports Knowledge accuracy, Practice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05132
  bibtex_key: hong2025towards
  confidence: high
---

# principlismqa-eval

> Towards Assessing Medical Ethics from Knowledge to Practice — Chang Hong et al. (2025) (arXiv:2508.05132, 2025)

## What this evaluates

Evaluates large language models' ability to reason about medical ethics using the Principlism framework (autonomy, non-maleficence, beneficence, justice). It probes both theoretical knowledge of ethical principles and their practical application to complex, open-ended clinical dilemmas.

## Datasets

- **PrinciplismQA** — total ?; splits: test (-1)

## Metrics

- `Knowledge accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions assessing theoretical understanding of medical ethics principles.
- `Practice score` **(primary)** — range: percent
  - Score on open-ended clinical dilemmas, evaluated via expert-validated checklists or LLM-as-a-judge (GPT-4o) scoring keypoint alignment.
- `Overall score` — range: percent
  - Arithmetic mean of Knowledge accuracy and Practice score.

## Input / output format

**Input**: Multiple-choice questions for knowledge assessment; open-ended clinical ethical dilemmas for practice assessment.

**Output**: Single response per model per question. For multiple-choice, a selected option. For open-ended, a free-text explanation.

## Scoring recipe

```python
# Knowledge
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
knowledge_acc = correct / len(golds)

# Practice
practice_scores = [score_response_for_dilemma(resp, checklist) for resp in responses]
practice_score = mean(practice_scores)

# Overall
overall = (knowledge_acc + practice_score) / 2
```

## Common pitfalls

- Models often score higher on Knowledge than Practice, revealing a 'knowledge-action gap' in applying principles to dynamic scenarios.
- Domain fine-tuning can improve Practice scores but may cause forgetting of key medical ethics knowledge, lowering Knowledge accuracy.
- LLMs tend to overemphasize autonomy and non-maleficence while struggling to dynamically apply beneficence when principles conflict.

## Evidence (verbatim from paper)

> The overall results of PrinciplismQA evaluation are summarized in Table[6]. Among general large reasoning models, o3 achieved the highest overall score, with 74.4% Knowledge accuracy, 80.7 Practice score, and an overall score of 77.5.

## Citation

```bibtex
@misc{hong2025towards,
  title={Towards Assessing Medical Ethics from Knowledge to Practice},
  author={Chang Hong et al. (2025)},
  year={2025},
  note={arXiv:2508.05132}
}
```

- arXiv: 2508.05132