Rgbt Ground Eval

---
name: rgbt-ground-eval
description: Evaluates multi-modal visual grounding capabilities by requiring models to localize objects in images using both RGB and thermal infrared (TIR) modalities guided by text queries. It specifically probes robustness under complex real-world conditions such as low-light environments, small object sizes, and diverse weather/illumination variations. Use when the user wants to benchmark on RGBT-Ground, or asks about evaluating this task. Reports Acc@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.24561
  bibtex_key: [REDACTED]
  confidence: high
---

# rgbt-ground-eval

> RGBT-Ground Benchmark: Visual Grounding Beyond RGB in Complex Real-World Scenarios — Tianyi Zhao et al. (2025) (arXiv:2512.24561, 2025)

## What this evaluates

Evaluates multi-modal visual grounding capabilities by requiring models to localize objects in images using both RGB and thermal infrared (TIR) modalities guided by text queries. It specifically probes robustness under complex real-world conditions such as low-light environments, small object sizes, and diverse weather/illumination variations.

## Datasets

- **RGBT-Ground** — total ?; splits: val (-1), test (-1), testA (-1), testB (-1), testC (-1)

## Metrics

- `Acc@0.5` **(primary)** — range: percent
  - Accuracy calculated as the fraction of instances where the Intersection-over-Union (IoU) between the predicted bounding box and the ground-truth bounding box exceeds 0.5.

## Input / output format

**Input**: Paired RGB and thermal infrared (TIR) images with a corresponding natural language text query describing the target object.

**Output**: A single bounding box coordinate (e.g., [x_min, y_min, x_max, y_max]) localizing the described object in the image.

## Scoring recipe

```python
correct = 0
for pred_box, gt_box in zip(predictions, ground_truths):
    iou = calculate_iou(pred_box, gt_box)
    if iou > 0.5:
        correct += 1
accuracy = (correct / len(predictions)) * 100
```

## Common pitfalls

- Zero-shot transfer from standard RGB datasets (e.g., RefCOCO, Flickr30K) fails drastically due to domain shift and the absence of TIR modality in pretraining data.
- Reporting only aggregate accuracy masks performance degradation on specific conditions; the protocol requires separate reporting for testA, testB, and testC to properly assess robustness to object size and lighting.
- Models must handle both RGB and TIR modalities; evaluating only on RGB or only on TIR without proper cross-modal adaptation yields misleadingly low results and violates the benchmark's multi-modal design.

## Evidence (verbatim from paper)

> Following the previous visual grounding paper*[chen2023advancing, xiao2024towards]*, we adopt the Acc@0.5 as the metrics, measuring the localization accuracy when the intersection-over-union (IoU) between the predicted and ground-truth bounding boxes exceeds 0.5. In addition, we report results separately for val, test, testA, testB, and testC subsets to evaluate model robustness under diverse conditions.

## Citation

```bibtex
@misc{zhao2025rgbtground,
  title={RGBT-Ground Benchmark: Visual Grounding Beyond RGB in Complex Real-World Scenarios},
  author={Tianyi Zhao et al. (2025)},
  year={2025},
  note={arXiv:2512.24561}
}
```

- arXiv: 2512.24561