Rjua Sps Clinical Eval

---
name: rjua-sps-clinical-eval
description: Evaluates LLMs' clinical capabilities across single-turn QA, diagnostic reasoning, and multi-turn dialogue using a urology-specific standardized patient dataset. It probes information gathering, diagnostic logic, treatment planning, and adherence to clinical workflows. Use when the user wants to benchmark on RJUA-SPs, or asks about evaluating this task. Reports Diagnosis Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.16446
  bibtex_key: liu2024towardsautomatic
  confidence: high
---

# rjua-sps-clinical-eval

> Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm — Liu et al. (2024) (arXiv:2403.16446, 2024)

## What this evaluates

Evaluates LLMs' clinical capabilities across single-turn QA, diagnostic reasoning, and multi-turn dialogue using a urology-specific standardized patient dataset. It probes information gathering, diagnostic logic, treatment planning, and adherence to clinical workflows.

## Datasets

- **RJUA-SPs** — total ?; splits: test (-1)

## Metrics

- `Diagnosis Accuracy` **(primary)** — range: percent
  - Percentage of correct diagnoses compared to ground-truth retrieved via RAE from Standardized Patients.
- `Treatment Accuracy` — range: percent
  - Percentage of correct treatment recommendations compared to ground-truth retrieved via RAE.
- `Diagnostic Reasoning Accuracy` — range: percent
  - Percentage of correctly recommended laboratory tests and examinations compared to ground-truth.
- `Clinical Capability Scores` — range: score
  - Scores for Information Completeness, Behavior Standardization, Guidance Rationality, Diagnostic Logicality, Treatment Logicality, and Clinical Applicability, calculated by matching dialogue turns against clinical practice pathways (LCP) via RAE.

## Input / output format

**Input**: Patient initial query or symptoms (single-turn); dialogue history with standardized patient responses (multi-turn).

**Output**: Model-generated medical advice, diagnosis, treatment recommendations, or dialogue turns simulating a doctor.

## Scoring recipe

```python
def score_model(model_output, task_type):
    gt = retrieve_ground_truth_from_SPs(task_type)
    if task_type in ['single_turn_qa', 'diagnostic_reasoning']:
        return 100.0 if exact_match(model_output, gt) else 0.0
    elif task_type == 'multi_turn_dialogue':
        capability_scores = {}
        for capability in ['Information Completeness', 'Guidance Rationality', 'Diagnostic Logicality', 'Treatment Logicality', 'Clinical Applicability']:
            matches = sum(1 for turn in model_output if match_clinic_pathway(turn, gt, capability))
            capability_scores[capability] = (matches / len(model_output)) * 100
        return capability_scores
    return 0.0
```

## Common pitfalls

- Models frequently hallucinate unnecessary medical tests not indicated by ground-truth clinical pathways.
- LLMs often fail to follow the standard clinical workflow order for collecting patient information, leading to low Behavior Standardization scores.
- Performance drops significantly in zero-shot settings where models lack complete medical context.

## Evidence (verbatim from paper)

> Single-turn Clinical QA. We calculate the diagnosis and treatment accuracy, where RAE can retrieve the ground-truth from SPs. Medical Diagnostic Reasoning. We calculate the accuracy of the recommended laboratory test and examination, where RAE can retrieve the ground-truth from SPs. Multi-turn Diagnostic Dialogue. Importantly, RAE can measure the clinical capabilities of LLMs by the retrieval task.

## Citation

```bibtex
@misc{liu2024towardsautomatic,
  title={Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2403.16446}
}
```

- arXiv: 2403.16446