Tunevlseg Eval

---
name: tunevlseg-eval
description: Evaluates the robustness and zero-shot adaptation of vision-language segmentation models under prompt tuning across diverse medical and natural domain datasets. It probes how different prompt tuning strategies and prompt depths handle domain shifts and varying class counts. Use when the user wants to benchmark on Kvasir-SEG, ClinicDB, BKAI, ISIC 2016, DFU 2022, CAMUS, BUSI, CheXlocalize, Cityscapes, PascalVOC, or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.05239
  bibtex_key: adhikari2024tunevlseg
  confidence: high
---

# tunevlseg-eval

> TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models — Adhikari et al. (2024) (arXiv:2410.05239, 2024)

## What this evaluates

Evaluates the robustness and zero-shot adaptation of vision-language segmentation models under prompt tuning across diverse medical and natural domain datasets. It probes how different prompt tuning strategies and prompt depths handle domain shifts and varying class counts.

## Datasets

- **Kvasir-SEG** — total ?; splits: train (800), val (100), test (100)
- **ClinicDB** — total ?; splits: train (490), val (61), test (61)
- **BKAI** — total ?; splits: train (800), val (100), test (100)
- **ISIC 2016** — total ?; splits: train (810), val (90), test (379)
- **DFU 2022** — total ?; splits: train (1600), val (200), test (200)
- **CAMUS** — total ?; splits: train (4800), val (600), test (600)
- **BUSI** — total ?; splits: train (624), val (78), test (78)
- **CheXlocalize** — total ?; splits: train (1279), val (446), test (452)
- **Cityscapes** — total ?; splits: train (34723), val (6005), test (-1)
- **PascalVOC** — total ?; splits: train (2170), val (2148), test (-1)

## Metrics

- `Dice score` **(primary)** — range: [0, 1]
  - Computed as 2 * |A ∩ B| / (|A| + |B|) between predicted and ground truth masks. The paper uses Dice Loss for training, implying Dice score for evaluation.

## Input / output format

**Input**: Resized images (416x416 for CRIS, 352x352 for CLIPSeg) with foreground class name(s) provided as text prompts. Images are normalized and lightly augmented during training.

**Output**: Binary or multi-class segmentation mask per image, corresponding to the foreground class(es) specified in the prompt.

## Scoring recipe

```python
def compute_dice(pred, gt):
    intersection = np.sum(pred * gt)
    union = np.sum(pred) + np.sum(gt)
    return (2.0 * intersection) / (union + 1e-6)
# Average Dice score across test set
```

## Common pitfalls

- Significant domain shift between medical and open-domain datasets requires careful prompt depth tuning.
- Hyperparameters are swept per dataset (20 runs each), leading to high variance across datasets.
- Medical datasets have few classes (1-10) while open-domain have ~20, affecting context learner performance.
- Only foreground class names are used for prompts, ignoring background or multi-class labels.

## Evidence (verbatim from paper)

> For our empirical analysis of medical datasets, we utilize eight of the datasets and their splits provided by Poudel et al. ... For the loss function, we used a combined loss of Dice Loss and Binary Cross Entropy loss.

## Citation

```bibtex
@misc{adhikari2024tunevlseg,
  title={TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models},
  author={Adhikari et al. (2024)},
  year={2024},
  note={arXiv:2410.05239}
}
```

- arXiv: 2410.05239