Graphwalker Eval

---
name: graphwalker-eval
description: Evaluates a graph-guided in-context learning framework for clinical reasoning on electronic health records. It probes the model's ability to select non-redundant, interacting demonstrations based on patient semantic graphs and information gain signals to improve prediction accuracy on mortality, length-of-stay, readmission, and clinical QA tasks. Use when the user wants to benchmark on MIMIC-III, MIMIC-IV, CMB, MedQA, CMB-clin, or asks about evaluating this task. Reports AUROC, AUPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.06684
  bibtex_key: fang2026graphwalker
  confidence: high
---

# graphwalker-eval

> GraphWalker: Graph-Guided In-Context Learning for Clinical Reasoning on Electronic Health Records — Yue Fang et al. (2026) (arXiv:2604.06684, 2026)

## What this evaluates

Evaluates a graph-guided in-context learning framework for clinical reasoning on electronic health records. It probes the model's ability to select non-redundant, interacting demonstrations based on patient semantic graphs and information gain signals to improve prediction accuracy on mortality, length-of-stay, readmission, and clinical QA tasks.

## Datasets

- **MIMIC-III** — total ?; splits: train (-1), val (-1), test (-1)
- **MIMIC-IV** — total ?; splits: train (-1), val (-1), test (-1)
- **CMB** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **CMB-clin** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.
- `AUPRC` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve. Evaluates model performance on imbalanced datasets by plotting precision against recall.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `ma-ROC` — range: [0, 1]
  - Macro-averaged ROC for discretized length-of-stay bins. Computes ROC per bin and averages them.
- `mi-ROC` — range: [0, 1]
  - Micro-averaged ROC for discretized length-of-stay bins. Aggregates contributions of all bins before computing ROC.
- `exact-match (EM)` — range: [0, 1]
  - Binary metric: 1 if the predicted answer exactly matches the ground truth, 0 otherwise. Averaged over the dataset.
- `BLEU-1 / BLEU-4` — range: [0, 100]
  - N-gram precision metric (1-gram and 4-gram) with brevity penalty, standard for text generation evaluation.
- `ROUGE-L` — range: [0, 100]
  - Recall-oriented overlap metric based on the longest common subsequence between predicted and reference text.

## Input / output format

**Input**: Patient-level EHR sequences (visit-sequenced) for prediction tasks; clinical questions/prompts for CMB, MedQA, and CMB-clin.

**Output**: Binary or categorical labels (e.g., mortality, readmission), discretized LOS bins, or free-text answers for clinical QA.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task in ['mortality', 'readmission']:
        return roc_auc_score(golds, predictions), \
               average_precision_score(golds, predictions), \
               f1_score(golds, predictions)
    elif task == 'los':
        bins = discretize(golds)
        return macro_roc_auc(bins, predictions), micro_roc_auc(bins, predictions)
    elif task in ['CMB', 'MedQA']:
        return np.mean([p == g for p, g in zip(predictions, golds)])
    elif task == 'CMB-clin':
        return bleu_1(predictions, golds), bleu_4(predictions, golds), rouge_l(predictions, golds)
```

## Common pitfalls

- Discretizing length-of-stay (LOS) labels into bins changes the evaluation distribution and requires careful bin definition to match established benchmarks.
- Using generic semantic embeddings instead of EHR-specific encoders for demonstration retrieval leads to significant performance drops due to lack of clinical nuance.
- Ignoring redundancy and interaction effects among demonstrations causes diminishing marginal returns in context-constrained settings.

## Evidence (verbatim from paper)

> For mortality and readmission prediction, we employ three widely used evaluation metrics, including the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and the F1-score. For LOS prediction, following Harutyunyan et al. (Harutyunyan et al., 2019), we discretize LOS labels into multiple bins and evaluate performance using macro-averaged ROC (ma-ROC) and micro-averaged ROC (mi-ROC).

## Citation

```bibtex
@misc{fang2026graphwalker,
  title={GraphWalker: Graph-Guided In-Context Learning for Clinical Reasoning on Electronic Health Records},
  author={Yue Fang et al. (2026)},
  year={2026},
  note={arXiv:2604.06684}
}
```

- arXiv: 2604.06684