Rsmeb Eval

---
name: rsmeb-eval
description: Evaluates a vision-language model's ability to perform zero-shot classification, cross-modal retrieval, visual question answering, and fine-grained spatial grounding (including region-caption retrieval and geo-localization) on remote sensing imagery. It measures how well instruction-conditioned contrastive pretraining aligns multimodal features with geospatial metadata and textual prompts. Use when the user wants to benchmark on AID, Million-AID, RSI-CB, EuroSAT, UCM, PatternNet, RSITMD, RSICD, UCM-caption, LRBEN, HRBEN, or asks about evaluating this task. Reports Friedman score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.11490
  bibtex_key: [REDACTED]
  confidence: high
---

# rsmeb-eval

> VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing — Emanuel Sánchez Aimar et al. (2025) (arXiv:2512.11490, 2025)

## What this evaluates

Evaluates a vision-language model's ability to perform zero-shot classification, cross-modal retrieval, visual question answering, and fine-grained spatial grounding (including region-caption retrieval and geo-localization) on remote sensing imagery. It measures how well instruction-conditioned contrastive pretraining aligns multimodal features with geospatial metadata and textual prompts.

## Datasets

- **AID** — total ?; splits: test (-1)
- **Million-AID** — total ?; splits: test (-1)
- **RSI-CB** — total ?; splits: test (-1)
- **EuroSAT** — total ?; splits: test (-1)
- **UCM** — total ?; splits: test (-1)
- **PatternNet** — total ?; splits: test (-1)
- **RSITMD** — total ?; splits: test (-1)
- **RSICD** — total ?; splits: test (-1)
- **UCM-caption** — total ?; splits: test (-1)
- **LRBEN** — total ?; splits: test (-1)
- **HRBEN** — total ?; splits: test (-1)

## Metrics

- `accuracy` — range: percent
  - Percentage of correctly classified images out of the total test set.
- `recall@1/5/10 (R@k)` — range: percent
  - Fraction of queries where the ground-truth item appears in the top-k retrieved results. The paper reports the average of R@1, R@5, and R@10 for cross-modal retrieval.
- `precision@1 (P@1)` — range: percent
  - Fraction of queries where the top-1 retrieved result matches the ground truth. Used for region-based CIR, VQA, visual grounding, spatial localization, and semantic geo-localization.
- `Friedman score` **(primary)** — range: other
  - Average rank of a method across all datasets/tasks in a meta-task. Lower scores indicate better overall performance.

## Input / output format

**Input**: Multimodal inputs including satellite images, text prompts/questions, bounding box coordinates, and geographic coordinates (latitude/longitude). Inputs are truncated to 4,096 tokens. For classification, a 20-prompt ensemble is used.

**Output**: Class labels for classification; ranked lists of retrieved images or texts for retrieval tasks; bounding boxes or image IDs for grounding/localization; and text answers for VQA.

## Scoring recipe

```python
def compute_metric(predictions, golds, metric_type, k=1):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type in ['recall', 'precision']:
        top_k = predictions[:k]
        return sum(1 for g in golds if g in top_k) / len(golds)
    elif metric_type == 'avg_recall':
        r1 = sum(1 for g in golds if g in predictions[:1]) / len(golds)
        r5 = sum(1 for g in golds if g in predictions[:5]) / len(golds)
        r10 = sum(1 for g in golds if g in predictions[:10]) / len(golds)
        return (r1 + r5 + r10) / 3
    elif metric_type == 'friedman':
        return sum(ranks) / len(ranks)
```

## Common pitfalls

- Evaluating specialized baselines (e.g., RemoteCLIP) in-distribution while testing the proposed model zero-shot, which inflates baseline scores.
- Using a single prompt for classification instead of the recommended 20-prompt ensemble, which significantly underestimates embedding model accuracy.
- Applying instruction prompts to dual-encoder baselines, which the authors note typically hurts their performance since they were not trained with instructions.

## Evidence (verbatim from paper)

> We evaluate each task using metrics according to the literature: accuracy for classification; recall@1/5/10 (R@k) and the average of the three metrics for cross‑modal retrieval *[[26], [45]]*; and precision@1 (P@1) for region-based CIR, VQA, visual grounding, spatial localization, and semantic geo‑localization *[[15]]*.

## Citation

```bibtex
@misc{sanchezaimar2025vlm2geovec,
  title={VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing},
  author={Emanuel Sánchez Aimar et al. (2025)},
  year={2025},
  note={arXiv:2512.11490}
}
```

- arXiv: 2512.11490