Referring Segmentation Eval

---
name: referring-segmentation-eval
description: Evaluates a model's ability to perform dense grounded understanding by localizing and segmenting specific objects in images and videos based on natural language instructions or referring expressions. Use when the user wants to benchmark on Ref-SAV, RefCOCO, RefCOCO+, RefCOCOg, MeVIS, Ref-YTVOS, ReVOS, or asks about evaluating this task. Reports cIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.04001
  bibtex_key: yuan2025sa2va
  confidence: high
---

# referring-segmentation-eval

> Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos — Haobo Yuan et al. (arXiv:2501.04001, 2025)

## What this evaluates

Evaluates a model's ability to perform dense grounded understanding by localizing and segmenting specific objects in images and videos based on natural language instructions or referring expressions.

## Datasets

- **Ref-SAV** — total 37000; splits: val (37000)
- **RefCOCO** — total 17000; splits: test (-1)
- **RefCOCO+** — total 17000; splits: test (-1)
- **RefCOCOg** — total 22000; splits: test (-1)
- **MeVIS** — total 600; splits: test (-1)
- **Ref-YTVOS** — total 3500; splits: test (-1)
- **ReVOS** — total 1700; splits: test (-1)

## Metrics

- `cIoU` **(primary)** — range: [0, 1]
  - Conditional Intersection over Union between the predicted segmentation mask and the ground-truth mask, conditioned on the referring expression.
- `J&F` — range: [0, 1]
  - Joint and F-measure, computed as the average of frame-wise IoU (J) and boundary F-score (F) across all video frames.

## Input / output format

**Input**: Image or video frames accompanied by a text prompt (referring expression or instruction).

**Output**: A segmentation mask (binary or probability map) for the referred object, generated by decoding the '[SEG]' token hidden state through SAM2's decoder.

## Scoring recipe

```python
def compute_cIoU(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    return intersection / union if union > 0 else 0.0

def compute_JF(pred_masks, gt_masks):
    J = np.mean([compute_cIoU(p, g) for p, g in zip(pred_masks, gt_masks)])
    F = compute_boundary_F1(pred_masks, gt_masks)
    return (J + F) / 2
```

## Common pitfalls

- cIoU conditions on the referring expression; evaluating without aligning the mask to the specific object mentioned in the prompt yields invalid scores.
- J&F requires strict frame-by-frame evaluation for videos; averaging masks across frames or ignoring temporal consistency will produce incorrect results.
- The mask must be decoded specifically from the '[SEG]' token's hidden state via SAM2's decoder; using alternative decoding heads will not match reported baselines.

## Evidence (verbatim from paper)

> For image referring segmentation, we adopt cIoU. For referring video object segmentation, we adopt J&F. ... the segmentation mask is obtained by decoding the hidden state of the “[SEG]” token through SAM2’s decoder.

## Citation

```bibtex
@misc{yuan2025sa2va,
  title={Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos},
  author={Haobo Yuan et al.},
  year={2025},
  note={arXiv:2501.04001}
}
```

- arXiv: 2501.04001