Rsvg Eval

---
name: rsvg-eval
description: This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes. Use when the user wants to benchmark on RSVGD, or asks about evaluating this task. Reports Pr@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.12634
  bibtex_key: zhan2022rsvg
  confidence: high
---

# rsvg-eval

> RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data — Zhan et al. (2022) (arXiv:2210.12634, 2022)

## What this evaluates

This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes.

## Datasets

- **RSVGD** — total 38320; splits: train (-1), val (-1), test (-1); repo https://github.com/ZhanYang-nwpu/RSVG-pytorch

## Metrics

- `Pr@0.5` **(primary)** — range: percent
  - Percentage of image-query pairs where the predicted bounding box has an Intersection-over-Union (IoU) with the ground-truth box greater than or equal to 0.5.
- `Pr@0.6` — range: percent
  - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.6.
- `Pr@0.7` — range: percent
  - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.7.
- `Pr@0.8` — range: percent
  - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.8.
- `Pr@0.9` — range: percent
  - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.9.
- `meanIoU` — range: [0, 1]
  - Average IoU across all test samples: (1/M) * sum(I_t / U_t), where I_t and U_t are the intersection and union areas for sample t, and M is the dataset size.
- `cumIoU` — range: [0, 1]
  - Cumulative IoU across all test samples: sum(I_t) / sum(U_t), aggregating intersection and union areas before division.

## Input / output format

**Input**: A remote sensing image and a natural language query describing a target object within the image.

**Output**: A single predicted bounding box (typically [x_min, y_min, x_max, y_max]) localizing the described object.

## Scoring recipe

```python
def score(predictions, gold_boxes):
    ious = [compute_iou(p, g) for p, g in zip(predictions, gold_boxes)]
    metrics = {}
    for thresh in [0.5, 0.6, 0.7, 0.8, 0.9]:
        metrics[f'Pr@{thresh}'] = sum(1 for i in ious if i >= thresh) / len(ious) * 100
    metrics['meanIoU'] = sum(ious) / len(ious)
    # cumIoU = sum(I_t) / sum(U_t). Since IoU = I_t/U_t, I_t = IoU * U_t.
    metrics['cumIoU'] = sum(iou * union_area(p, g) for iou, p, g in zip(ious, predictions, gold_boxes)) / \
                        sum(union_area(p, g) for p, g in zip(predictions, gold_boxes))
    return metrics
```

## Common pitfalls

- IoU thresholds are strict in remote sensing due to small object sizes; Pr@0.9 often drops significantly compared to Pr@0.5.
- Cluttered backgrounds and scale variations in satellite imagery frequently cause false positives, making meanIoU and cumIoU more informative than binary precision at high thresholds.
- The dataset split is randomized by expression (40/10/50), not by image, which can lead to data leakage if the same image appears in train and test with different queries.

## Evidence (verbatim from paper)

> Given an RS image-query pair, the predicted bounding box is considered right if the intersection-over-union (IoU) with the ground-truth bounding box is above a threshold. In previous visual grounding works, a threshold of 0.5 is used as an accuracy metric. We report the metrics with IoU thresholds at 0.5, 0.6, 0.7, 0.8, and 0.9, termed as Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, and Pr@0.9, respectively. In addition, we follow the evaluation metrics of [[59]], including mean IoU and cumulative IoU (cumIoU), with the following equations: meanIoU = 1/M sum(I_t/U_t), and cumIoU = (sum I_t)/(sum U_t).

## Citation

```bibtex
@misc{zhan2022rsvg,
  title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data},
  author={Zhan et al. (2022)},
  year={2022},
  note={arXiv:2210.12634}
}
```

- arXiv: 2210.12634