Pneumonia Xray Zero Shot Eval

---
name: pneumonia-xray-zero-shot-eval
description: This benchmark evaluates the zero-shot diagnostic capability of vision-language models on chest X-ray images for binary pneumonia detection. It probes whether models can accurately classify radiological findings without task-specific fine-tuning, relying instead on prompt engineering and pre-trained visual reasoning. Use when the user wants to benchmark on Chest radiographic Images (Pneumonia), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21839
  bibtex_key: prahallad2025evaluating
  confidence: medium
---

# pneumonia-xray-zero-shot-eval

> Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images — Prahallad et al. (2025) (arXiv:2510.21839, 2025)

## What this evaluates

This benchmark evaluates the zero-shot diagnostic capability of vision-language models on chest X-ray images for binary pneumonia detection. It probes whether models can accurately classify radiological findings without task-specific fine-tuning, relying instead on prompt engineering and pre-trained visual reasoning.

## Datasets

- **Chest radiographic Images (Pneumonia)** — total 5863; splits: train (5216), test (624), val (16)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the number of correctly classified images divided by the total number of test images. For binary classification, it represents the proportion of predictions matching the ground truth labels (Normal or Pneumonia).

## Input / output format

**Input**: Raw chest X-ray image files from the test set.

**Output**: Binary classification label: either 'Normal' or 'Pneumonia'.

## Scoring recipe

```python
def calculate_accuracy(predictions, ground_truth):
    correct = sum(1 for pred, true in zip(predictions, ground_truth) if pred == true)
    return correct / len(ground_truth)
```

## Common pitfalls

- The dataset exclusively contains pediatric patients (ages 1-5), so results do not generalize to adult chest X-rays.
- Zero-shot performance is highly sensitive to prompt design; varying prompt styles (e.g., reasoning-based vs. concise) drastically changes accuracy.
- The test set is imbalanced (390 Pneumonia vs. 234 Normal), which can skew accuracy if not accounted for.

## Evidence (verbatim from paper)

> This study evaluates OpenAI’s gpt-4o model in zero-shot classification of chest X-rays for pneumonia using four prompt designs, revealing that concise, feature-focused prompts achieve 74% accuracy—outperforming reasoning-based prompts—highlighting the critical role of prompt engineering in visual-language model performance for medical diagnosis.

## Citation

```bibtex
@misc{prahallad2025evaluating,
  title={Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images},
  author={Prahallad et al. (2025)},
  year={2025},
  note={arXiv:2510.21839}
}
```

- arXiv: 2510.21839