Raddiagseg Eval

---
name: raddiagseg-eval
description: Evaluates a vision-language model's ability to perform joint radiological diagnosis, abnormality detection, and multi-target segmentation on X-ray and CT images. It probes the model's capacity for open-ended visual question answering, precise pixel-level mask generation, and robustness to label-imbalanced medical data. Use when the user wants to benchmark on RadDiagSeg-D, VQA-RAD, SLAKE, or asks about evaluating this task. Reports F1, Dice.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.18188
  bibtex_key: li2025raddiagsegm
  confidence: high
---

# raddiagseg-eval

> RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology — Chengrun Li et al. (2025) (arXiv:2510.18188, 2025)

## What this evaluates

Evaluates a vision-language model's ability to perform joint radiological diagnosis, abnormality detection, and multi-target segmentation on X-ray and CT images. It probes the model's capacity for open-ended visual question answering, precise pixel-level mask generation, and robustness to label-imbalanced medical data.

## Datasets

- **RadDiagSeg-D** — total 28800; splits: train (22000), test (6800)
- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for VQA, detection, and diagnosis tasks.
- `Dice` **(primary)** — range: [0, 1]
  - Dice coefficient measuring overlap between predicted and ground-truth masks: 2 * |A ∩ B| / (|A| + |B|). Used for segmentation quality.
- `Recall` — range: [0, 1]
  - True positive rate: TP / (TP + FN). Used for VQA and detection tasks.
- `OpenQ-Acc` — range: [0, 1]
  - Exact match accuracy for open-ended question answers.

## Input / output format

**Input**: Radiological image (X-ray or CT slice) paired with a text prompt/question specifying the task (e.g., diagnosis query, detection instruction, or referring segmentation prompt).

**Output**: Text response (diagnosis label, detection yes/no, or open-ended answer) followed by segmentation mask tokens/coordinates. For segmentation tasks, pixel-level binary masks are generated.

## Scoring recipe

```python
def score(predictions, gold):
    f1s, recalls, dices = [], [], []
    for pred, g in zip(predictions, gold):
        pred_tok, gold_tok = set(pred[0].split()), set(g[0].split())
        if not pred_tok: f1s.append(0.0); recalls.append(0.0)
        else:
            prec = len(pred_tok & gold_tok) / len(pred_tok)
            rec = len(pred_tok & gold_tok) / len(gold_tok)
            f1s.append(2 * prec * rec / (prec + rec + 1e-6))
            recalls.append(rec)
        mask_p, mask_g = pred[1], g[1]
        inter = np.sum(mask_p & mask_g)
        dices.append(2 * inter / (np.sum(mask_p) + np.sum(mask_g) + 1e-6))
    return {'F1': np.mean(f1s), 'Recall': np.mean(recalls), 'Dice': np.mean(dices)}
```

## Common pitfalls

- Label imbalance in RadDiagSeg-D necessitates using F1 instead of accuracy for diagnosis/detection; accuracy would be misleading.
- For joint VQA-Seg tasks, baseline models often fail to generate both text and masks, requiring task amputation or resulting in empty evaluation fields.
- 3D modality slices (CT/MRI) from the same volume must not leak across train/test splits to prevent data leakage and inflated performance.

## Evidence (verbatim from paper)

> For evaluation, we adopt F1 and Recall as metrics for the VQA tasks. We additionally document the Recall and Accuracy for open-ended questions. Following common practices, the Dice score is used to benchmark the quality of segmentation. For RadDiagSeg-D, given the label imbalance, we use F1 as the metric for detection and diagnosis.

## Citation

```bibtex
@misc{li2025raddiagsegm,
  title={RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology},
  author={Chengrun Li et al. (2025)},
  year={2025},
  note={arXiv:2510.18188}
}
```

- arXiv: 2510.18188