Dreamlip Zero Shot Eval

---
name: dreamlip-zero-shot-eval
description: Evaluates the zero-shot transfer capability of language-image pre-trained models across image-text retrieval, semantic segmentation, image classification, and vision-language reasoning tasks. Use when the user wants to benchmark on ImageNet, MSCOCO, Flickr30K, ADE20K-150, VOC-20, or asks about evaluating this task. Reports R@K, Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17007
  bibtex_key: zheng2024dreamlip
  confidence: high
---

# dreamlip-zero-shot-eval

> DreamLIP: Language-Image Pre-training with Long Captions — Kecheng Zheng et al. (arXiv:2403.17007, 2024)

## What this evaluates

Evaluates the zero-shot transfer capability of language-image pre-trained models across image-text retrieval, semantic segmentation, image classification, and vision-language reasoning tasks.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **MSCOCO** — total ?; splits: test (-1)
- **Flickr30K** — total ?; splits: test (-1)
- **ADE20K-150** — total ?; splits: test (-1)
- **VOC-20** — total ?; splits: test (-1)

## Metrics

- `R@K` **(primary)** — range: percent
  - Recall at top-K retrieved items. Computed by ranking texts/images by similarity score and checking if the ground truth appears in the top-K.
- `Top-1 accuracy` **(primary)** — range: percent
  - Percentage of images correctly classified by selecting the class label with the highest cosine similarity to the image embedding.
- `mIOU` — range: percent
  - Mean Intersection over Union across all semantic classes, measuring segmentation mask overlap with ground truth.

## Input / output format

**Input**: Image resized to 224x224. Text truncated or padded to 77 tokens. For zero-shot classification, text inputs are predefined prompts containing class label names.

**Output**: Class label prediction (classification), ranked list of retrieved items (retrieval), pixel-wise class labels (segmentation).

## Scoring recipe

```python
def evaluate_zero_shot(image, text_prompts, gold_labels):
    img_emb = model.encode_image(image)
    txt_embs = model.encode_text(text_prompts)
    sim = (img_emb @ txt_embs.T) / temperature
    pred = text_prompts[torch.argmax(sim, dim=1)]
    return (pred == gold_labels).float().mean()

def evaluate_retrieval(img_embs, txt_embs, gold_indices, k):
    sim_matrix = img_embs @ txt_embs.T
    ranks = torch.argsort(sim_matrix, descending=True)
    recall = (ranks[:, :k].argmax(dim=1) == gold_indices).float().mean()
    return recall
```

## Common pitfalls

- Using custom prompt templates instead of CLIP's predefined class-label prompts changes zero-shot results significantly.
- Text inputs must be strictly truncated/padded to 77 tokens to match the pre-trained text encoder's capacity.
- Evaluating semantic segmentation requires fine-tuning on COCO-stuff, not just zero-shot inference.

## Evidence (verbatim from paper)

> Following CLIP, we select 11 visual recognition datasets under the zero-shot setting... The same zero-shot classification protocol is applied following [47], which uses predefined prompts as text inputs. We use R@K to report the recall of top-K retrieval items. Top-1 accuracy is used for evaluation.

## Citation

```bibtex
@misc{zheng2024dreamlip,
  title={DreamLIP: Language-Image Pre-training with Long Captions},
  author={Kecheng Zheng et al.},
  year={2024},
  note={arXiv:2403.17007}
}
```

- arXiv: 2403.17007