Rscc Change Caption Eval

---
name: rscc-change-caption-eval
description: Evaluates vision-language models' ability to generate detailed, semantically accurate captions describing changes between bi-temporal remote sensing image pairs, particularly in disaster scenarios. It probes spatiotemporal reasoning, fine-grained environmental change detection, and long-text generation quality. Use when the user wants to benchmark on RSCC, or asks about evaluating this task. Reports ST5-SCS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.01907
  bibtex_key: chen2025rscc
  confidence: high
---

# rscc-change-caption-eval

> RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Events — Chen et al. (2025) (arXiv:2509.01907, 2025)

## What this evaluates

Evaluates vision-language models' ability to generate detailed, semantically accurate captions describing changes between bi-temporal remote sensing image pairs, particularly in disaster scenarios. It probes spatiotemporal reasoning, fine-grained environmental change detection, and long-text generation quality.

## Datasets

- **RSCC** — total 62351; splits: test (-1); repo https://github.com/Bili-Sakura/RSCC

## Metrics

- `ST5-SCS` **(primary)** — range: percent
  - Sentence T5-XXL Embedding with Sharpened Cosine Similarity. Computes cosine similarity between T5-XXL embeddings of the reference and hypothesis captions, applying a sharpening function with parameters q=0 and p=3.
- `ROUGE` — range: percent
  - Standard n-gram overlap metric measuring recall/precision of n-grams between reference and generated captions.
- `METEOR` — range: percent
  - N-gram overlap metric that incorporates synonymy, stemming, and word order to measure semantic similarity.

## Input / output format

**Input**: Bi-temporal remote sensing image pairs (pre- and post-disaster). Optionally augmented with building damage masks/info from the xBD dataset.

**Output**: Natural language caption describing the observed changes between the two images.

## Scoring recipe

```python
def compute_st5_scs(predictions, references, q=0, p=3):
    ref_embeds = get_t5_xxl_embedding(references)
    pred_embeds = get_t5_xxl_embedding(predictions)
    cos_sim = cosine_similarity(ref_embeds, pred_embeds)
    sharpened = (cos_sim + q) ** p
    return np.mean(sharpened) * 100
```

## Common pitfalls

- N-gram overlap metrics (ROUGE, METEOR) are suboptimal for measuring semantic similarity in long captions.
- Some models suffer from severe output repetition, artificially inflating caption length and degrading scores.
- Scaling correction decoding strategies (VCD, DoLa, DeCo) do not consistently improve performance on this complex spatiotemporal reasoning task.

## Evidence (verbatim from paper)

> For model evaluation, we compare the text similarity with n-gram overlap metrics including ROUGE and METEOR. While the aforementioned measures are commonly reported in image captioning works, we find they are suboptimal to measure the semantic similarity across long texts. Therefore, we follow Kaggle LLM Prompt Recovery Competition and introduce Sentence T5-XXL Embedding with Sharpened Cosine Similarity (ST5-SCS) to get a well-established similarity measure. We set q=0 and p=3 for sharpened cosine similarity.

## Citation

```bibtex
@misc{chen2025rscc,
  title={RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Events},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2509.01907}
}
```

- arXiv: 2509.01907