Schain Medical Reasoning Eval

---
name: schain-medical-reasoning-eval
description: Evaluates medical vision-language models on disease classification and structured visual reasoning. It probes the model's ability to localize lesions, generate clinically faithful chain-of-thought rationales, and produce accurate diagnostic classifications grounded in visual evidence. Use when the user wants to benchmark on S-Chain, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22728
  bibtex_key: leduc2025schain
  confidence: high
---

# schain-medical-reasoning-eval

> S-Chain: Structured Visual Chain-of-Thought For Medicine — Khai Le-Duc et al. (2025) (arXiv:2510.22728, 2025)

## What this evaluates

Evaluates medical vision-language models on disease classification and structured visual reasoning. It probes the model's ability to localize lesions, generate clinically faithful chain-of-thought rationales, and produce accurate diagnostic classifications grounded in visual evidence.

## Datasets

- **S-Chain** — total 12000; splits: train (10783), test (1542)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly predicted disease classes in the final diagnostic task (Q4).
- `F1` — range: [0, 1]
  - F1 score for disease classification (Q4), calculated to capture class balance across diagnostic categories.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union for bounding box localization (Q1), measuring overlap between predicted and ground-truth lesion regions.
- `BLEU / METEOR / BERTScore` — range: [0, 1]
  - Standard text similarity metrics used to evaluate the semantic faithfulness and clinical plausibility of generated chain-of-thought descriptions (Q2).
- `Lesion Grading Accuracy` — range: [0, 1]
  - Accuracy of predicted lesion severity grades against expert-annotated scores (Q3).

## Input / output format

**Input**: Medical image paired with a structured question (Q1-Q4). For RAG settings, top-k retrieved medical passages are concatenated. For controlled experiments, ground-truth intermediate steps (bounding boxes, CoT text, grading) may be provided as input.

**Output**: For Q1: bounding box coordinates. For Q2: chain-of-thought text description. For Q3: lesion grade score. For Q4: final diagnostic classification label.

## Scoring recipe

```python
def score(predictions, gold):
    # Q4 Classification
    acc = sum(1 for p, g in zip(predictions['q4'], gold['q4']) if p == g) / len(gold['q4'])
    f1 = f1_score(gold['q4'], predictions['q4'], average='macro')
    # Q1 Localization
    miou = mean_iou(predictions['q1_boxes'], gold['q1_boxes'])
    # Q2 Text
    bleu = bleu_score(predictions['q2_text'], gold['q2_text'])
    meteor = meteor_score(predictions['q2_text'], gold['q2_text'])
    bertscore = bertscore_f1(predictions['q2_text'], gold['q2_text'])
    # Q3 Grading
    grade_acc = sum(1 for p, g in zip(predictions['q3'], gold['q3']) if p == g) / len(gold['q3'])
    return {'Accuracy': acc, 'F1': f1, 'mIoU': miou, 'BLEU': bleu, 'METEOR': meteor, 'BERTScore': bertscore, 'Grading_Acc': grade_acc}
```

## Common pitfalls

- High final diagnostic accuracy may mask unfaithful or hallucinated reasoning traces that ignore visual evidence.
- Synthetic CoT training often inherits teacher model hallucinations, leading to missing or misaligned bounding boxes.
- Textual supervision of bounding box coordinates yields weaker reasoning alignment compared to visual prompting (highlighted ROIs).

## Evidence (verbatim from paper)

> Task and Metrics: We evaluate models primarily on disease classification (Q4), reporting both Accuracy and F1 to capture overall correctness and class balance. Intermediate steps are also assessed: bounding box localization (Q1) with mIoU, lesion grading (Q3) with Accuracy against expert scores, and CoT descriptions (Q2) with BLEU, METEOR, BERTScore for semantic similarity for faithfulness and clinical plausibility.

## Citation

```bibtex
@misc{leduc2025schain,
  title={S-Chain: Structured Visual Chain-of-Thought For Medicine},
  author={Khai Le-Duc et al. (2025)},
  year={2025},
  note={arXiv:2510.22728}
}
```

- arXiv: 2510.22728