Xlrs Bench Eval

---
name: xlrs-bench-eval
description: Evaluates multimodal large language models on ultra-high-resolution remote sensing imagery using vision-language question answering. It probes both perception (e.g., object classification, counting, spatial relations) and reasoning capabilities across various sub-tasks. Use when the user wants to benchmark on XLRS-Bench, LRS-VQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21375
  bibtex_key: [REDACTED]
  confidence: high
---

# xlrs-bench-eval

> GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution — Wang et al. (2025) (arXiv:2505.21375, 2025)

## What this evaluates

Evaluates multimodal large language models on ultra-high-resolution remote sensing imagery using vision-language question answering. It probes both perception (e.g., object classification, counting, spatial relations) and reasoning capabilities across various sub-tasks.

## Datasets

- **XLRS-Bench** — total ?; splits: test (-1)
- **LRS-VQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For XLRS-Bench, accuracy is computed per sub-task (L-1 dimension) and averaged across all sub-tasks. For LRS-VQA, accuracy is computed after converting open-ended QA to multiple-choice format.

## Input / output format

**Input**: Ultra-high-resolution remote sensing images (up to 8K resolution) paired with zero-shot, uniform text prompts/questions.

**Output**: Text responses to VQA questions. For LRS-VQA, responses are matched against multiple-choice options generated by GPT-4o.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The paper explicitly reports only the L-1 dimension accuracy for XLRS-Bench; L-3 and L-4 results are in the appendix and should not be used as the primary metric.
- LRS-VQA evaluation requires converting open-ended answers to multiple-choice using an external LLM (GPT-4o), which may introduce format bias or hallucination in distractor generation.
- All models must be evaluated in a zero-shot setting with uniform prompts to ensure fair comparison.

## Evidence (verbatim from paper)

> Following XLRS-Bench*[xlrs-bench]*, we evaluated the accuracy and reported of L-1 dimension for the VQA task, with L-3 and L-4 results available in the appendix. GeoLLaVA-8K achieved 56.28% accuracy, outperforming LLaVA-Next*[llava-next]* (55.07%) by 1.21%, confirming strong generalization to new datasets and formats.

## Citation

```bibtex
@misc{wang2025geollava8k,
  title={GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.21375}
}
```

- arXiv: 2505.21375