Rs Vlm Zero Shot Eval

---
name: rs-vlm-zero-shot-eval
description: Evaluates the zero-shot image classification and text-to-image retrieval capabilities of Vision-Language Models (VLMs) fine-tuned on remote sensing data. It probes the model's ability to generalize to unseen RS scenes and text queries without task-specific fine-tuning, while also measuring resistance to catastrophic forgetting on general-domain benchmarks. Use when the user wants to benchmark on AID, EuroSAT, fMoW, Million-AID, PatternNet, RESISC, RSI-CB, ImageNet-1K, UCM Captions, RSICD, RSITMD, or asks about evaluating this task. Reports zero-shot top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14744
  bibtex_key: ge2024rsteller
  confidence: high
---

# rs-vlm-zero-shot-eval

> RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models — Junyao Ge et al. (arXiv:2408.14744, 2024)

## What this evaluates

Evaluates the zero-shot image classification and text-to-image retrieval capabilities of Vision-Language Models (VLMs) fine-tuned on remote sensing data. It probes the model's ability to generalize to unseen RS scenes and text queries without task-specific fine-tuning, while also measuring resistance to catastrophic forgetting on general-domain benchmarks.

## Datasets

- **AID** — total 2000; splits: test (2000)
- **EuroSAT** — total 2700; splits: test (2700)
- **fMoW** — total 106081; splits: val (106081)
- **Million-AID** — total 10000; splits: train (10000)
- **PatternNet** — total 30400; splits: train (30400)
- **RESISC** — total 31500; splits: train (31500)
- **RSI-CB** — total 24747; splits: train (24747)
- **ImageNet-1K** — total 50000; splits: val (50000)
- **UCM Captions** — total 210; splits: test (210)
- **RSICD** — total 1093; splits: test (1093)
- **RSITMD** — total 452; splits: test (452)

## Metrics

- `zero-shot top-1 accuracy` **(primary)** — range: percent
  - Percentage of images correctly classified into their ground-truth category using the model's zero-shot classifier (typically via cosine similarity between image and text embeddings).
- `zero-shot retrieval recall@K` — range: percent
  - Recall at K (K=1, 5, 10), measuring the percentage of queries where the correct matching image or text appears within the top K results of the ranked list.

## Input / output format

**Input**: Image patch (or full image) and corresponding text caption/query.

**Output**: Predicted class label (classification) or ranked list of images/texts (retrieval).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, queries, gold_images, K=10):
    # Classification
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    accuracy = (correct / len(gold_labels)) * 100
    
    # Retrieval
    recalls = []
    for query, gold_img in zip(queries, gold_images):
        scores = model.encode(query) @ model.encode(images).T
        top_k_indices = np.argsort(scores)[-K:][::-1]
        recalls.append(1 if gold_img in top_k_indices else 0)
    recall_at_k = (sum(recalls) / len(queries)) * 100
    return accuracy, recall_at_k
```

## Common pitfalls

- The paper uses non-standard splits for some benchmarks (e.g., fMoW on 'val', Million-AID/PatternNet/RESISC/RSI-CB on 'train') which differs from typical zero-shot evaluation protocols.
- Performance heavily depends on the initial CLIP checkpoint used for continual pre-training; results are not directly comparable across different base checkpoints without accounting for their distinct zero-shot baselines.
- Catastrophic forgetting is evaluated on ImageNet-1K separately from RS benchmarks, requiring careful tracking of general-domain performance degradation.

## Evidence (verbatim from paper)

> For evaluative purposes, we employ a series of zero-shot classification and image retrieval tasks. For zero-shot classification, we use eight widely recognized and comparatively extensive benchmark datasets, including AID, EuroSAT, fMoW, Million-AID, PatternNet, NWPU-RESISC45 (RESISC), RSI-CB256 (RSI-CB), and ImageNet-1K. ... The evaluation results are presented in Tables [9] and [10]. These tables report the top-1 accuracy for zero-shot classification tasks, as well as the average recall rates at top-1 (R@1), top-5 (R@5), and top-10 (R@10) for zero-shot text-to-image retrieval tasks.

## Citation

```bibtex
@misc{ge2024rsteller,
  title={RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models},
  author={Junyao Ge et al.},
  year={2024},
  note={arXiv:2408.14744}
}
```

- arXiv: 2408.14744