Ct Rate Zero Shot Eval

---
name: ct-rate-zero-shot-eval
description: This benchmark evaluates the zero-shot multi-abnormality detection capability of a visual-language foundation model on 3D chest CT volumes. It probes the model's ability to generalize to unseen data distributions and classify multiple pathologies simultaneously without task-specific supervised training. Use when the user wants to benchmark on CT-RATE, RAD-ChestCT, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17834
  bibtex_key: hamamci2024developing
  confidence: high
---

# ct-rate-zero-shot-eval

> Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography — Hamamci et al. (2024) (arXiv:2403.17834, 2024)

## What this evaluates

This benchmark evaluates the zero-shot multi-abnormality detection capability of a visual-language foundation model on 3D chest CT volumes. It probes the model's ability to generalize to unseen data distributions and classify multiple pathologies simultaneously without task-specific supervised training.

## Datasets

- **CT-RATE** — total 50188; splits: train (-1), val (-1); repo https://github.com/ibrahimethemhamamci/CT-CLIP
- **RAD-ChestCT** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, computed per abnormality and averaged across all 18 abnormalities.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall, computed per abnormality and averaged.
- `accuracy` — range: [0, 1]
  - Proportion of correctly classified instances per abnormality, averaged across abnormalities.
- `precision` — range: [0, 1]
  - Proportion of true positive predictions among all positive predictions per abnormality, averaged.

## Input / output format

**Input**: 3D chest CT volume (reconstructed from DICOM slices)

**Output**: Softmax probabilities for each of the 18 predefined abnormalities, indicating the likelihood of presence.

## Scoring recipe

```python
for each abnormality in 18:
    pos_logit = cosine_similarity(volume_emb, positive_prompt_emb)
    neg_logit = cosine_similarity(volume_emb, negative_prompt_emb)
    logits = [pos_logit, neg_logit]
    probs = softmax(logits)
    pred_prob = probs[0]  # positive logit probability
    pred_label = 1 if pred_prob > threshold else 0
    compute AUROC, F1, accuracy, precision against ground truth
average metrics across all 18 abnormalities
```

## Common pitfalls

- Treating the task as single-label classification instead of multi-label, as medical CTs often contain multiple co-occurring abnormalities.
- Ignoring prompt engineering variations, which significantly impact zero-shot classification performance.
- Failing to account for distribution shift by only evaluating on the internal validation set without the external RAD-ChestCT set.

## Evidence (verbatim from paper)

> Our zero-shot approach yields significantly higher mean AUROC (0.099 higher in internal validation and 0.082 higher in external validation), mean F1 score (0.061 higher in internal validation and 0.052 higher in external validation), mean accuracy (0.07 higher in internal validation and 0.052 higher in external validation), and mean precision (0.065 higher in internal validation and 0.047 higher in external validation) compared to the supervised baseline.

## Citation

```bibtex
@misc{hamamci2024developing,
  title={Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography},
  author={Hamamci et al. (2024)},
  year={2024},
  note={arXiv:2403.17834}
}
```

- arXiv: 2403.17834