Dynamicare Medical Diagnosis Eval

---
name: dynamicare-medical-diagnosis-eval
description: Evaluates a dynamic multi-agent framework's ability to perform interactive, open-ended medical diagnosis by querying patients and ranking potential diagnoses. It also assesses the system's performance on interactive multiple-choice medical QA and the quality of simulated patient responses. Use when the user wants to benchmark on MIMIC-Patient, MEDIQ, or asks about evaluating this task. Reports Hit@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.02616
  bibtex_key: shang2025dynamicare
  confidence: high
---

# dynamicare-medical-diagnosis-eval

> DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making — Shang et al. (2025) (arXiv:2507.02616, 2025)

## What this evaluates

Evaluates a dynamic multi-agent framework's ability to perform interactive, open-ended medical diagnosis by querying patients and ranking potential diagnoses. It also assesses the system's performance on interactive multiple-choice medical QA and the quality of simulated patient responses.

## Datasets

- **MIMIC-Patient** — total 500; splits: test (500)
- **MEDIQ** — total 340; splits: test (340)

## Metrics

- `Hit@K` **(primary)** — range: [0, 1]
  - Binary indicator (1 if at least one ground-truth diagnosis appears in the top-K predicted diagnoses, 0 otherwise). Averaged over the dataset.
- `Rec@K` — range: [0, 1]
  - Proportion of all ground-truth diagnoses found within the top-K predicted diagnoses. Averaged over the dataset.
- `Accuracy` — range: [0, 1]
  - Exact match rate for multiple-choice questions on the MEDIQ benchmark.
- `Ave-Q` — range: other
  - Average number of questions asked by the doctor agent per patient case.
- `Truthfulness` — range: [0, 2]
  - Human-annotated score (0-2 scale) measuring consistency between patient responses and the ground-truth JSON record.
- `Relevance` — range: [0, 2]
  - Human-annotated score (0-2 scale) measuring how directly and adequately the patient response addresses the doctor's question.

## Input / output format

**Input**: Interactive dialogue history with patient records or static JSON patient profiles; for MEDIQ, multiple-choice clinical questions.

**Output**: Ranked list of up to 10 predicted diagnoses (mapped to ICD-9 codes); for MEDIQ, a single selected option; for Patient System, a natural language response.

## Scoring recipe

```python
def score_hit_rec(predictions, ground_truth):
    # predictions: list of up to 10 predicted ICD-9 codes
    # ground_truth: list of ground truth ICD-9 codes
    # Match rule: first 3 digits must match
    pred_3digit = set(code[:3] for code in predictions)
    gt_3digit = set(code[:3] for code in ground_truth)
    hit = 1.0 if pred_3digit & gt_3digit else 0.0
    rec = len(pred_3digit & gt_3digit) / len(gt_3digit) if gt_3digit else 0.0
    return hit, rec
```

## Common pitfalls

- ICD-9 matching uses only the first 3 digits (high-level category), not full 5-digit codes, which significantly affects Hit/Rec scores.
- Ground truth typically contains 1-5 correct labels per patient, so Hit@K and Rec@K behave differently than in standard single-label retrieval benchmarks.
- MEDIQ evaluation uses only 200 randomly sampled cases from iMedQA, not the full dataset, so results are not directly comparable to full-dataset baselines.

## Evidence (verbatim from paper)

> We then evaluate the results using the top-k hit rate (Hit@5, Hit@10) and recall (Rec@5, Rec@10), which measure the system’s ability to rank correct diagnoses among its predictions. As the doctor agent is prompted to return a list of up to 10 likely diagnoses, and the ground truth typically includes between 1 to 5 correct labels, Hit@K metrics help assess how well the system captures correct diagnoses within a ranked list, whereas Rec@K offers a perspective on how many ground truth diagnoses are successfully identified.

## Citation

```bibtex
@misc{shang2025dynamicare,
  title={DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making},
  author={Shang et al. (2025)},
  year={2025},
  note={arXiv:2507.02616}
}
```

- arXiv: 2507.02616