Medsyn Eval

---
name: medsyn-eval
description: Evaluates multimodal large language models on their ability to generate differential diagnoses (DDx) and select final diagnoses (FDx) for complex clinical cases. It probes cross-modal evidence calibration, testing how models weigh textual versus visual clinical evidence, and measures their sensitivity to specific evidence types. Use when the user wants to benchmark on MEDSYN, or asks about evaluating this task. Reports FDx SelectionAcc. (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.21950
  bibtex_key: chen2026medsyn
  confidence: high
---

# medsyn-eval

> MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models — Chen et al. (2026) (arXiv:2602.21950, 2026)

## What this evaluates

Evaluates multimodal large language models on their ability to generate differential diagnoses (DDx) and select final diagnoses (FDx) for complex clinical cases. It probes cross-modal evidence calibration, testing how models weigh textual versus visual clinical evidence, and measures their sensitivity to specific evidence types.

## Datasets

- **MEDSYN** — total ?; splits: English (-1), Chinese (-1)

## Metrics

- `DDx CoverageRate (%)` — range: percent
  - Percentage of cases where the model's generated list of differential diagnoses fully covers the ground truth set of potential diagnoses.
- `FDx SelectionAcc. (%)` **(primary)** — range: percent
  - Percentage of cases where the model correctly selects the single ground-truth final diagnosis from the available options.
- `Evidence Sensitivity` — range: [0, 1]
  - Jensen-Shannon divergence between the model's answer posterior with all clinical evidence and the posterior after removing a specific evidence type: JSD(p(y|E) || p(y|E\{e_m})).

## Input / output format

**Input**: Multimodal clinical case containing up to 7 distinct clinical evidence types (e.g., medical history text, physical findings, raw diagnostic images like CT/microscopy, lab results). Evaluated in a zero-shot setting.

**Output**: For DDx: a list of potential diagnoses. For FDx: a single selected final diagnosis.

## Scoring recipe

```python
# DDx CoverageRate
correct_ddx = sum(1 for c in cases if set(gold_ddx[c]).issubset(set(pred_ddx[c])))
ddx_rate = correct_ddx / len(cases) * 100

# FDx SelectionAcc
correct_fdx = sum(1 for c in cases if pred_fdx[c] == gold_fdx[c])
fdx_acc = correct_fdx / len(cases) * 100
```

## Common pitfalls

- Confusing DDx coverage (enumerating possibilities) with FDx accuracy (selecting the single correct answer), as models show a ~20pp gap between them.
- Assuming textual evidence is always more helpful; ablations show removing or randomizing text can actually improve FDx accuracy due to model bias toward text over discriminative visual evidence.
- Cross-modal misalignment: identical evidence presented as raw images vs. expert text yields different model sensitivities, often clustering off the identity line.

## Evidence (verbatim from paper)

> DDx CoverageRate (%) | FDx SelectionAcc. (%)

## Citation

```bibtex
@misc{chen2026medsyn,
  title={MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.21950}
}
```

- arXiv: 2602.21950