Drugcareqa Eval

---
name: drugcareqa-eval
description: Evaluates an AI system's ability to perform integrated clinical decision-making by simulating real-world online medical consultations. It probes the model's capacity to reason through patient symptoms, generate accurate diagnoses, and recommend appropriate medications within a unified workflow. Use when the user wants to benchmark on DrugCareQA, or asks about evaluating this task. Reports diagnostic and medication recommendation accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.10461
  bibtex_key: zheng2025medcoact
  confidence: high
---

# drugcareqa-eval

> MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision — Zheng et al. (2025) (arXiv:2510.10461, 2025)

## What this evaluates

Evaluates an AI system's ability to perform integrated clinical decision-making by simulating real-world online medical consultations. It probes the model's capacity to reason through patient symptoms, generate accurate diagnoses, and recommend appropriate medications within a unified workflow.

## Datasets

- **DrugCareQA** — total 2700; splits: test (-1)

## Metrics

- `diagnostic and medication recommendation accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's predicted diagnosis and medication recommendation both exactly match the expert-verified ground truth. Calculated as (correct predictions / total instances) * 100.

## Input / output format

**Input**: Patient chief complaint or multi-turn dialogue describing symptoms, potentially including demographic and clinical presentation details.

**Output**: A structured response containing a final diagnosis and a specific medication recommendation.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if pred['diagnosis'] == gold_item['diagnosis'] and pred['medication'] == gold_item['medication']:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models may optimize for diagnosis only and ignore medication recommendation, or vice versa, since the benchmark requires both to be correct for a positive score.
- The dataset uses real-world patient dialogues with ambiguous symptom descriptions, which can cause models to hallucinate diagnoses if they rely too heavily on rigid multiple-choice formats from other benchmarks.
- Medication recommendations must match the exact drug names specified in the expert-verified ground truth, as similar drug distractors are intentionally included to test clinical logic.

## Evidence (verbatim from paper)

> We integrate diagnostic reasoning with medication selection into a unified evaluation framework. The dataset encompasses 2,700 annotated cases across seven clinical departments with dual quality verification combining knowledge base validation and expert review. Models are evaluated on diagnostic and medication recommendation accuracy, achieving 67.58% on DrugCareQA—a benchmark of 2,700 real-world patient-doctor dialogues that unifies diagnosis and treatment in a clinically realistic workflow.

## Citation

```bibtex
@misc{zheng2025medcoact,
  title={MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.10461}
}
```

- arXiv: 2510.10461