Vgphrasecut Eval

---
name: vgphrasecut-eval
description: This benchmark evaluates language-based image segmentation by requiring models to ground natural language phrases into precise image regions. It probes a model's ability to handle long-tail categories, attributes, relationships, and varying object sizes in open-vocabulary settings. Use when the user wants to benchmark on VGPhraseCut, or asks about evaluating this task. Reports mean-IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2008.01187
  bibtex_key: wu2020phrasecut
  confidence: high
---

# vgphrasecut-eval

> PhraseCut: Language-based Image Segmentation in the Wild — Chenyun Wu et al. (2020) (arXiv:2008.01187, 2020)

## What this evaluates

This benchmark evaluates language-based image segmentation by requiring models to ground natural language phrases into precise image regions. It probes a model's ability to handle long-tail categories, attributes, relationships, and varying object sizes in open-vocabulary settings.

## Datasets

- **VGPhraseCut** — total ?; splits: test (-1)

## Metrics

- `mean-IoU` **(primary)** — range: [0, 1]
  - Average Intersection over Union across all test instances. IoU is computed as the area of overlap between the predicted and ground-truth segmentation masks divided by the area of their union.
- `cum-IoU` — range: [0, 1]
  - Cumulative Intersection over Union, calculated as the total intersection area across all instances divided by the total union area. Dominated by large targets.
- `Pr@0.5` — range: [0, 1]
  - Precision at IoU threshold 0.5, measuring the fraction of predictions with IoU ≥ 0.5.

## Input / output format

**Input**: An input image and a natural language referring expression (phrase) specifying a target object or region.

**Output**: A binary segmentation mask (or region proposal) corresponding to the target described by the phrase.

## Scoring recipe

```python
def compute_mean_iou(predictions, ground_truths):
    ious = []
    for pred, gt in zip(predictions, ground_truths):
        intersection = np.logical_and(pred, gt).sum()
        union = np.logical_or(pred, gt).sum()
        ious.append(intersection / union if union > 0 else 0.0)
    return np.mean(ious)
```

## Common pitfalls

- RMI achieves high cum-IoU but low mean-IoU because it performs well on large targets but fails on small ones, skewing cumulative metrics.
- MattNet's evaluation relies on ground-truth instance boxes for context; using predicted detections instead causes a severe performance drop, highlighting dataset sparsity issues.
- Performance varies drastically across category frequency subsets (top 100 vs 500+), so reporting only overall mean-IoU masks long-tail failures.

## Evidence (verbatim from paper)

> RMI obtains high cum-IoU but low mean-IoU scores because it handles large targets well but fails on small ones (see Table 4 “small/mid/large” subsets). cum-IoU is dominated by large targets while our dataset many small targets: 20.2% of our data has the target region smaller than 2% of the image area, while the smallest target in RefCOCO is 2.4% of the image.

## Citation

```bibtex
@misc{wu2020phrasecut,
  title={PhraseCut: Language-based Image Segmentation in the Wild},
  author={Chenyun Wu et al. (2020)},
  year={2020},
  note={arXiv:2008.01187}
}
```

- arXiv: 2008.01187