Rsrcc Eval

---
name: rsrcc-eval
description: This benchmark evaluates large language models' ability to perform fine-grained, region-specific semantic reasoning on remote sensing image pairs. It probes localized change comprehension by asking models to answer binary, multiple-choice, and open-ended questions about specific changes (e.g., new construction, vegetation loss) within satellite imagery. Use when the user wants to benchmark on RSRCC, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20623
  bibtex_key: kazoom2026rsrcc
  confidence: high
---

# rsrcc-eval

> RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking — Kazoom et al. (2026) (arXiv:2604.20623, 2026)

## What this evaluates

This benchmark evaluates large language models' ability to perform fine-grained, region-specific semantic reasoning on remote sensing image pairs. It probes localized change comprehension by asking models to answer binary, multiple-choice, and open-ended questions about specific changes (e.g., new construction, vegetation loss) within satellite imagery.

## Datasets

- **RSRCC** — total ?; splits: test (-1)

## Metrics

- `Human Agreement (%)` — range: percent
  - Percentage of human evaluators who agree with the generated questions or answers.
- `Accuracy (%)` **(primary)** — range: percent
  - Correctness rate for binary (Yes/No) and multiple-choice responses.
- `BLEU` — range: percent
  - n-gram precision measuring lexical overlap between model-generated answers and ground-truth answers.
- `BERTScore (F1)` — range: [0, 1]
  - Semantic similarity score between model-generated and human-written captions using contextual embeddings.
- `CIDEr` — range: percent
  - Consensus-based metric measuring similarity between generated captions and multiple human references using TF-IDF weighted n-grams.
- `SPICE` — range: percent
  - Semantic propositional metric evaluating agreement between generated and reference captions at the level of objects, attributes, and relations.

## Input / output format

**Input**: Paired high-resolution satellite images (typically 512x512 px) with a region-specific question (Yes/No, multiple-choice, or open-ended) targeting localized changes.

**Output**: For closed-ended questions: 'Yes'/'No' or a selected option. For open-ended questions: a free-text caption or answer.

## Scoring recipe

```python
if question_type in ['Yes/No', 'Multiple-Choice']:
    score = 1.0 if prediction == gold else 0.0
    return mean(score)
elif question_type == 'Open-Ended':
    bleu = compute_bleu(prediction, gold)
    bertscore = compute_bertscore_f1(prediction, gold)
    cider = compute_cider(prediction, gold_references)
    spice = compute_spice(prediction, gold_references)
    return {'BLEU': bleu, 'BERTScore': bertscore, 'CIDEr': cider, 'SPICE': spice}
```

## Common pitfalls

- Accuracy is only applicable to closed-ended (Yes/No, Multiple-Choice) questions, not open-ended ones.
- BLEU, CIDEr, and SPICE are strictly for open-ended responses and should not be computed for binary/multiple-choice outputs.
- Human Agreement is evaluated per dataset creation pipeline stage, not just as a final model benchmark metric.

## Evidence (verbatim from paper)

> We evaluate our framework both quantitatively and qualitatively to assess scalability, reliability, and semantic consistency using the following metrics: Human Agreement (%) - percentage of evaluators agreeing with generated questions or answers. Accuracy (%) - correctness of binary (Yes/No) and multiple-choice responses. BLEU - $n$-gram precision measuring lexical overlap between model-generated answers and the ground-truth answers in the dataset. BERTScore (F1) - semantic similarity between model-generated and human-written captions.

## Citation

```bibtex
@misc{kazoom2026rsrcc,
  title={RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking},
  author={Kazoom et al. (2026)},
  year={2026},
  note={arXiv:2604.20623}
}
```

- arXiv: 2604.20623