Medical Emr Extraction Eval

---
name: medical-emr-extraction-eval
description: Probes a model's ability to extract structured clinical information from unstructured physician-patient consultation dialogues. It evaluates how accurately the model maps conversational text into predefined medical record fields such as chief complaint, diagnosis, and treatment recommendations. Use when the user wants to benchmark on EMRModel Dataset, or asks about evaluating this task. Reports weighted average F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.16448
  bibtex_key: zhao2025emrmodel
  confidence: high
---

# medical-emr-extraction-eval

> EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records — Zhao et al. (2025) (arXiv:2504.16448, 2025)

## What this evaluates

Probes a model's ability to extract structured clinical information from unstructured physician-patient consultation dialogues. It evaluates how accurately the model maps conversational text into predefined medical record fields such as chief complaint, diagnosis, and treatment recommendations.

## Datasets

- **EMRModel Dataset** — total 8665; splits: train (7329), val (800), test (536)

## Metrics

- `weighted average F1 score` **(primary)** — range: [0, 1]
  - Calculates the F1 score for each target field, then computes a weighted average where weights are the character counts of each field in the ground truth. The formula is F1_overall = sum(w_f * F1_f) / sum(w_f) for total weight > 0, else 0.

## Input / output format

**Input**: Unstructured text of physician-patient consultation dialogues (transcribed from audio recordings).

**Output**: Structured medical record template containing predefined fields (e.g., age, gender, chief complaint, past medical history, preliminary diagnosis, treatment recommendations).

## Scoring recipe

```python
def compute_weighted_f1(preds, golds, fields):
    total_weighted_f1 = 0.0
    total_weight = 0.0
    for f in fields:
        pred_val = preds.get(f, "")
        gold_val = golds.get(f, "")
        f1_val = f1_score(gold_val, pred_val)
        w_f = len(gold_val)
        total_weighted_f1 += w_f * f1_val
        total_weight += w_f
    return total_weighted_f1 / total_weight if total_weight > 0 else 0.0
```

## Common pitfalls

- Using a simple arithmetic mean across fields instead of the character-weighted average, which misrepresents the importance of longer clinical fields.
- Treating missing or empty model outputs inconsistently; the protocol explicitly states missing fields should be treated as empty strings for F1 calculation.
- Ignoring the sample standard deviation of F1 scores, which is required to assess performance stability across different test samples.

## Evidence (verbatim from paper)

> To comprehensively and accurately evaluate the performance of the LLM on the task of generating structured medical records from unstructured text, this study employs the weighted average F1 score as the primary evaluation metric. To assess the model’s performance stability across different samples, we further calculate the sample standard deviation of the weighted average F1 scores obtained for each individual sample. This standard deviation reflects the variability in the model’s performance when processing different test samples. A lower standard deviation indicates more stable model performance.

## Citation

```bibtex
@misc{zhao2025emrmodel,
  title={EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2504.16448}
}
```

- arXiv: 2504.16448