Referring Image Segmentation Eval

---
name: referring-image-segmentation-eval
description: Evaluates visual grounding capabilities by measuring how well a model segments or localizes objects in images based on natural language descriptions. It probes the model's ability to handle ambiguous references, diverse textual forms, and generalized referring expressions without task-specific decoders. Use when the user wants to benchmark on RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05123
  bibtex_key: yu2025latent
  confidence: high
---

# referring-image-segmentation-eval

> Latent Expression Generation for Referring Image Segmentation and Grounding — Yu et al. (2025) (arXiv:2508.05123, 2025)

## What this evaluates

Evaluates visual grounding capabilities by measuring how well a model segments or localizes objects in images based on natural language descriptions. It probes the model's ability to handle ambiguous references, diverse textual forms, and generalized referring expressions without task-specific decoders.

## Datasets

- **RefCOCO** — total ?; splits: val (-1), testA (-1), testB (-1)
- **RefCOCO+** — total ?; splits: val (-1), testA (-1), testB (-1)
- **RefCOCOg** — total ?; splits: val (-1), test (-1)
- **gRefCOCO** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union; computed as the average IoU across all samples.
- `oIoU` — range: [0, 1]
  - Overall IoU; computed by dividing the total intersection by the total union over all samples.
- `Prec@X` — range: [0, 1]
  - Percentage of samples where the predicted mask IoU exceeds a given threshold X (e.g., 0.5, 0.7, 0.9).
- `N-acc` — range: [0, 1]
  - No-target accuracy for GRES; defined as TP / (TP + FN), where TP correctly predicts no target and FN incorrectly predicts a target.

## Input / output format

**Input**: RGB image (resized to 480×480) paired with a natural language referring expression.

**Output**: Binary segmentation mask (generated by thresholding model output at 0.35).

## Scoring recipe

```python
def compute_metrics(pred_masks, gt_masks, no_target_idx=None):
    ious = [iou(p, g) for p, g in zip(pred_masks, gt_masks)]
    mIoU = sum(ious) / len(ious)
    oIoU = sum(inter(p, g) for p, g in zip(pred_masks, gt_masks)) / sum(union(p, g) for p, g in zip(pred_masks, gt_masks))
    prec_50 = sum(1 for i in ious if i > 0.5) / len(ious)
    prec_70 = sum(1 for i in ious if i > 0.7) / len(ious)
    prec_90 = sum(1 for i in ious if i > 0.9) / len(ious)
    n_acc = 0.0
    if no_target_idx is not None:
        tp = sum(1 for i in no_target_idx if ious[i] == 0)
        fn = sum(1 for i in no_target_idx if ious[i] > 0)
        n_acc = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return mIoU, oIoU, prec_50, prec_70, prec_90, n_acc
```

## Common pitfalls

- Confusing mIoU (per-sample average) with oIoU (global intersection over global union), which yield different values on imbalanced datasets.
- N-acc inverts standard detection logic: a True Positive means the model correctly predicts the absence of a target (IoU=0), not a successful localization.
- Prec@X thresholds are applied independently per sample, not as a global ranking cutoff.

## Evidence (verbatim from paper)

> For RIS, we use the standard metrics: mIoU, oIoU, and prec@X; mIoU (mean IoU) is the average IoU across all samples, while oIoU (overall IoU) divides the total intersection by the total union over all samples. Prec@X is the percentage of samples with IoU above threshold X. For REC, we employ Prec@50 (Acc.), which is an accuracy only when IoU is over 0.5. For GRES, in addition to mIoU and oIoU, we also compute N-acc (No-target accuracy) to evaluate the accuracy on no-target samples. N-acc is defined as $\frac{TP}{TP+FN}$, where a true positive (TP) indicates correctly predicting no target, otherwise a false negative (FN).

## Citation

```bibtex
@misc{yu2025latent,
  title={Latent Expression Generation for Referring Image Segmentation and Grounding},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2508.05123}
}
```

- arXiv: 2508.05123