Emsqa Eval

---
name: emsqa-eval
description: Evaluates large language models and retrieval-augmented generation systems on emergency medical services (EMS) multiple-choice questions across different clinical subject areas and certification levels. It probes the models' ability to apply domain-specific expertise and reasoning to answer standardized medical certification questions. Use when the user wants to benchmark on EMSQA, or asks about evaluating this task. Reports exact-match accuracy (Acc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.10900
  bibtex_key: ge2025expertguided
  confidence: high
---

# emsqa-eval

> Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering — Ge et al. (2025) (arXiv:2511.10900, 2025)

## What this evaluates

Evaluates large language models and retrieval-augmented generation systems on emergency medical services (EMS) multiple-choice questions across different clinical subject areas and certification levels. It probes the models' ability to apply domain-specific expertise and reasoning to answer standardized medical certification questions.

## Datasets

- **EMSQA** — total ?; splits: Public (-1), Private (-1)

## Metrics

- `exact-match accuracy (Acc)` **(primary)** — range: percent
  - Percentage of samples where the model's predicted answer exactly matches the ground-truth answer.
- `sample-based F1` — range: percent
  - F1 score computed per sample (treating answers as sets of labels due to multiple correct answers per question), then averaged across all samples.

## Input / output format

**Input**: Multiple-choice medical question, optionally accompanied by retrieved clinical documents (KB) and patient records (PR) for RAG methods, or provided with few-shot examples/prompting instructions.

**Output**: Predicted answer(s) to the multiple-choice question (single or multiple correct options).

## Scoring recipe

```python
def score(predictions, golds):
    exact_matches = 0
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_set = set(pred)
        gold_set = set(gold)
        if pred_set == gold_set:
            exact_matches += 1
        tp = len(pred_set & gold_set)
        fp = len(pred_set - gold_set)
        fn = len(gold_set - pred_set)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
        f1_scores.append(f1)
    acc = exact_matches / len(predictions) * 100
    f1 = sum(f1_scores) / len(f1_scores) * 100
    return {'Acc': acc, 'F1': f1}
```

## Common pitfalls

- Questions may have multiple correct answers, so exact-match requires predicting all correct options, not just one.
- The dataset is split into Public and Private subsets; performance should be reported separately to avoid data leakage or unfair comparison.
- RAG baselines use different retrieval corpora (KB, PR, or both), so comparing vanilla RAG to ExpertRAG requires matching the retrieval setup.

## Evidence (verbatim from paper)

> Since some questions in EMSQA have multiple correct answers, we report both exact-match accuracy (Acc) and sample-based F1*(Khashabi et al. [2018])*.

## Citation

```bibtex
@misc{ge2025expertguided,
  title={Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering},
  author={Ge et al. (2025)},
  year={2025},
  note={arXiv:2511.10900}
}
```

- arXiv: 2511.10900