Polish Medical Qa Eval

---
name: polish-medical-qa-eval
description: Evaluates large language models on Polish medical licensing and specialization exams to assess cross-lingual medical knowledge transfer, domain-specific understanding, and specialty-level accuracy compared to human medical graduates. Use when the user wants to benchmark on Polish Medical Exams (LEK/LDEK/PES), or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.00559
  bibtex_key: grzybowski2024polishenglishmedical
  confidence: high
---

# polish-medical-qa-eval

> Polish-English medical knowledge transfer: A new benchmark and results — Grzybowski et al. (2024) (arXiv:2412.00559, 2024)

## What this evaluates

Evaluates large language models on Polish medical licensing and specialization exams to assess cross-lingual medical knowledge transfer, domain-specific understanding, and specialty-level accuracy compared to human medical graduates.

## Datasets

- **Polish Medical Exams (LEK/LDEK/PES)** — total 1961; splits: LEK (977), LDEK (984), PES (-1)

## Metrics

- `score` **(primary)** — range: other
  - Total number of correctly answered multiple-choice questions. Model scores are compared against human test-taker distributions (mean ± standard deviation) and percentile ranks.

## Input / output format

**Input**: Polish-language multiple-choice medical exam questions with answer options.

**Output**: Model's selected answer option for each question.

## Scoring recipe

```python
def compute_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct

def rank_vs_human(model_score, human_scores):
    p25, p50, p75 = np.percentile(human_scores, [25, 50, 75])
    if model_score < min(human_scores): return 'below_min'
    elif model_score < p25: return 'p0_25'
    elif model_score < p50: return 'p25_50'
    elif model_score < p75: return 'p50_75'
    elif model_score < max(human_scores): return 'p75_max'
    else: return 'above_max'
```

## Common pitfalls

- Assuming exam score distributions are perfectly normal, which the authors note is an assumption rather than a proven fact.
- Small sample sizes in certain PES specializations (e.g., 6-9 participants) can heavily skew percentile comparisons and human baselines.
- Confusing cross-lingual translation fidelity with actual domain-specific medical knowledge retention.

## Evidence (verbatim from paper)

> The exams were taken by 33,929 participants for LEK and 4,366 for LDEK, totaling 38,295 results from medical graduates and final-year students in Poland. While all selected models pass the chosen LEK exams, only Meta-Llama-3.1-70B-Instruct and gpt-4o-2024-08-06 score within the range defined by an average number of points ± standard deviation achieved by humans.

## Citation

```bibtex
@misc{grzybowski2024polishenglishmedical,
  title={Polish-English medical knowledge transfer: A new benchmark and results},
  author={Grzybowski et al. (2024)},
  year={2024},
  note={arXiv:2412.00559}
}
```

- arXiv: 2412.00559