X Pcr Eval

---
name: x-pcr-eval
description: Evaluates multi-modal large language models on progressive clinical reasoning in ophthalmic diagnosis. It tests the model's ability to perform a six-stage diagnostic chain (from image quality assessment to clinical decision-making) while integrating cross-modality imaging data and calibrating its uncertainty. Use when the user wants to benchmark on X-PCR, or asks about evaluating this task. Reports Stage-Wise Accuracy (SWA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20350
  bibtex_key: wang2026xpcr
  confidence: high
---

# x-pcr-eval

> X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis — Gui Wang et al. (arXiv:2604.20350, 2026)

## What this evaluates

Evaluates multi-modal large language models on progressive clinical reasoning in ophthalmic diagnosis. It tests the model's ability to perform a six-stage diagnostic chain (from image quality assessment to clinical decision-making) while integrating cross-modality imaging data and calibrating its uncertainty.

## Datasets

- **X-PCR** — total 177868; splits: test (177868); repo https://github.com/CVI-SZU/X-PCR

## Metrics

- `Stage-Wise Accuracy (SWA)` **(primary)** — range: percent
  - Percentage of correct answers at each of the six reasoning stages (IQA, AL, LC, DD, SG, CD).
- `Chain Completion Rate (CCR)` — range: percent
  - Percentage of instances where the model correctly answers all six stages in the reasoning chain.
- `Expertise-Stratified Accuracy (ESA)` — range: percent
  - Accuracy computed separately for three clinical difficulty tiers: Resident (R), Attending (A), and Specialist (S).
- `Uncertainty-Aware Score (UAS)` — range: other
  - Weighted score aggregating confidence-accuracy categories (CC, CU, IU, IC) with difficulty and clinical impact weights to reward calibrated confidence and penalize overconfident errors.
- `Expected Calibration Error (ECE)` — range: other
  - Bin-wise discrepancy between the model's self-reported confidence and its empirical accuracy across confidence bins.

## Input / output format

**Input**: Multi-modal ophthalmic images (up to 6 modalities: EP, CFP, FFA, ICGA, OCT, RetCam) paired with clinical vignettes and stage-specific multiple-choice questions.

**Output**: Multiple-choice answer selection, self-reported confidence score [0,1], and step-by-step reasoning for each of the six clinical stages.

## Scoring recipe

```python
def compute_metrics(predictions, gold, confidences, difficulties):
    swa = {s: sum(p==g for p,g in zip(preds[s], gold[s]))/len(gold[s]) for s in stages}
    ccr = sum(all(p==g for p,g in zip(preds[s], gold[s])) for _ in instances)/len(instances)
    esa = {t: sum(p==g for p,g,d in zip(preds,gold,diff) if d==t)/count(t) for t in tiers}
    uas = sum(weight(cat(p,g,c))*impact(d) for p,g,c,d in zip(preds,gold,conf,diff))
    ece = sum(bin_size*abs(bin_acc-bin_conf) for bin in bins)
    return swa, ccr, esa, uas, ece
```

## Common pitfalls

- Evaluating stages in isolation ignores the benchmark's core dependency-aware design, masking error propagation across the diagnostic chain.
- Reporting only standard accuracy overlooks the uncertainty calibration component; models frequently exhibit Incorrect Confident (IC) behavior, which heavily penalizes UAS.
- Failing to stratify results by difficulty tier (R/A/S) and clinical impact weights violates the expertise-aware assessment protocol.

## Evidence (verbatim from paper)

> We define the following evaluation metrics: 1) Stage-Wise Accuracy (SWA) per reasoning stage; 2) Chain Completion Rate (CCR) for fully correct six-stage chains; 3) Expertise-Stratified Accuracy (ESA) within each difficulty tier. ... Model responses include self-reported confidence (normalized to $[0,1]$), categorized as Correct Confident (CC), Correct Uncertain (CU), Incorrect Uncertain (IU), or Incorrect Confident (IC). Each category receives a base score, aggregated with difficulty–impact weights into a weighted Uncertainty-Aware Score (UAS), rewarding calibrated confidence and penalizing overconfident errors. We further report Expected Calibration Error (ECE)...

## Citation

```bibtex
@misc{wang2026xpcr,
  title={X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis},
  author={Gui Wang et al.},
  year={2026},
  note={arXiv:2604.20350}
}
```

- arXiv: 2604.20350