Radgpt Tumor Report Eval

---
name: radgpt-tumor-report-eval
description: Evaluates an AI pipeline's ability to generate clinically accurate radiology reports from 3D CT scans by detecting tumors, measuring their size, localizing them within organ sub-segments, and staging cancers. It also assesses the textual similarity and diagnostic utility of generated reports compared to ground-truth clinical notes. Use when the user wants to benchmark on AbdomenAtlas 3.0, or asks about evaluating this task. Reports Tumor Detection Sensitivity & Specificity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.04678
  bibtex_key: bassi2025radgpt
  confidence: high
---

# radgpt-tumor-report-eval

> RadGPT: Constructing 3D Image-Text Tumor Datasets — Bassi et al. (2025) (arXiv:2501.04678, 2025)

## What this evaluates

Evaluates an AI pipeline's ability to generate clinically accurate radiology reports from 3D CT scans by detecting tumors, measuring their size, localizing them within organ sub-segments, and staging cancers. It also assesses the textual similarity and diagnostic utility of generated reports compared to ground-truth clinical notes.

## Datasets

- **AbdomenAtlas 3.0** — total ?; splits: test (-1); repo https://github.com/MrGiovanni/RadGPT

## Metrics

- `Tumor Detection Sensitivity & Specificity` **(primary)** — range: [0, 1]
  - Calculated via a zero-shot LLM (Llama-3.1) that classifies whether a generated report indicates tumor presence. Sensitivity = TP / (TP + FN); Specificity = TN / (TN + FP). Validated against radiologist labels with 96% LLM accuracy.
- `Tumor Size Measurement Accuracy` — range: percent
  - Manual evaluation by a radiologist. A reported measurement is correct if it deviates by ≤10% from the radiologist's ground-truth measurement using WHO standards.
- `Cancer Staging Accuracy` — range: percent
  - Percentage of correctly predicted T stages (T1–T3) for pancreatic adenocarcinoma on a private dataset (N=42).
- `Text Similarity (BLEU, METEOR, ROUGE-1/2/L, BERTScore)` — range: [0, 100]
  - Standard n-gram and embedding-based overlap metrics comparing generated reports to ground-truth clinical notes. BLEU, METEOR, ROUGE, and BERTScore are computed per standard implementations.

## Input / output format

**Input**: 3D CT scan volumes

**Output**: Radiology reports (structured or narrative) containing tumor presence, size, location, attenuation, volume, and cancer staging (e.g., T1–T4)

## Scoring recipe

```python
```python
# LLM-based detection sensitivity/specificity
reports = model.generate(ct_scans)
llm_preds = [llm.predict_tumor_present(r) for r in reports]
sensitivity = sum(p == 1 for p, g in zip(llm_preds, gt_labels) if g == 1) / sum(gt_labels)
specificity = sum(p == 0 for p, g in zip(llm_preds, gt_labels) if g == 0) / sum(1 - g for g in gt_labels)

# Manual size accuracy (subset)
correct = sum(1 for a, g in zip(ai_sizes, gt_sizes) if abs(a - g) / g <= 0.10)
size_accuracy = correct / len(ai_sizes)
```
```

## Common pitfalls

- Style variations (structured vs. narrative) heavily impact ROUGE and METEOR scores despite identical diagnostic content.
- Automatically matching AI-generated tumor measurements to ground-truth reports is difficult when multiple tumors are mentioned in the same sub-segment.
- LLM-based evaluation must be validated against radiologists, as zero-shot LLMs can hallucinate or misinterpret clinical phrasing without careful prompting.

## Evidence (verbatim from paper)

> RadGPT achieved 75.6% tumor detection precision and 93.8% tumor size measurement accuracy, according to manual evaluation. A radiologist evaluated the reports RadGPT created for 23 CT scans from UCSF. A reported tumor measurement was considered correct if it deviated by 10% or less from the radiologist's measurement (both use the WHO measuring standard [47]).

## Citation

```bibtex
@misc{bassi2025radgpt,
  title={RadGPT: Constructing 3D Image-Text Tumor Datasets},
  author={Bassi et al. (2025)},
  year={2025},
  note={arXiv:2501.04678}
}
```

- arXiv: 2501.04678