Lrs Vqa Eval

---
name: lrs-vqa-eval
description: Evaluates large vision-language models on high-resolution remote sensing imagery by testing their ability to answer questions about color, count, position, and open-ended descriptions. It probes the model's perception and reasoning capabilities on complex, large-scale satellite/aerial images. Use when the user wants to benchmark on MME-RealWorld-RS, LRS-VQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07588
  bibtex_key: luo2025lrs_vqa
  confidence: high
---

# lrs-vqa-eval

> When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning — Junwei Luo et al. (2025) (arXiv:2503.07588, 2025)

## What this evaluates

Evaluates large vision-language models on high-resolution remote sensing imagery by testing their ability to answer questions about color, count, position, and open-ended descriptions. It probes the model's perception and reasoning capabilities on complex, large-scale satellite/aerial images.

## Datasets

- **MME-RealWorld-RS** — total 1298; splits: test (1298)
- **LRS-VQA** — total 7333; splits: test (7333)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the percentage of correctly answered questions. For multiple-choice (color, count, position), exact match is used. For short open-ended answers, a WordNet-based semantic similarity score is computed, and the answer is considered correct if similarity ≥ 0.8.

## Input / output format

**Input**: High-resolution remote sensing image (up to 8064×8064 pixels or dynamically selected tiles) paired with a natural language question.

**Output**: Natural language text answer (short open-ended or multiple-choice option).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
        elif wordnet_similarity(pred, gold) >= 0.8:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Vicuna-1.5-based models tend to respond with only option A; the prompt must be modified by removing 'The best answer is:' to get valid outputs.
- Open-ended answers are not scored via exact string matching; they require WordNet-based semantic similarity (threshold ≥ 0.8).
- Localization recall (Table 7) only counts as successful if retained vision tokens cover >50% of the ground-truth region.

## Evidence (verbatim from paper)

> Benchmarks and evaluation metrics: i) MME-RealWorld-RS: the RS part of MME-Realworld[[91]], containing 1,298 RSIs with expert-annotated questions in three types: color, count, and position. We follow the official evaluation script but modify the prompt by removing “The best answer is:" to address Vicuna-1.5-based models’ tendency to respond with only option A. ii) LRS-VQA: it consists of 3 parts: LRS-FAIR, LRS-Bridge, and LRS-STAR, containing 2,272, 1,062, and 3,999 QA pairs, respectively. For the short open-ended format, we adopt a structured evaluation metric following[[22], [71]], using WordNet[[52]], with a semantic similarity threshold of 0.8.

## Citation

```bibtex
@misc{luo2025lrs_vqa,
  title={When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning},
  author={Junwei Luo et al. (2025)},
  year={2025},
  note={arXiv:2503.07588}
}
```

- arXiv: 2503.07588