Refcocom Eval

---
name: refcocom-eval
description: Evaluates a model's ability to perform referring expression segmentation at both object and part levels. It probes fine-grained cross-modal alignment and pixel-level semantic understanding by requiring precise mask prediction for diverse textual references. Use when the user wants to benchmark on RefCOCOm, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.01954
  bibtex_key: liu2025refcocom
  confidence: high
---

# refcocom-eval

> Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities — Jing Liu et al. (2025) (arXiv:2504.01954, 2025)

## What this evaluates

Evaluates a model's ability to perform referring expression segmentation at both object and part levels. It probes fine-grained cross-modal alignment and pixel-level semantic understanding by requiring precise mask prediction for diverse textual references.

## Datasets

- **RefCOCOm** — total 92000; splits: validation (-1), testA (-1), testB (-1); repo https://github.com/Rubics-Xuan/MRES

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection-over-Union between predicted and ground-truth binary segmentation masks. Computed as the average of |pred ∩ gt| / |pred ∪ gt| across all references.

## Input / output format

**Input**: An image paired with a natural language referring expression.

**Output**: A binary segmentation mask corresponding to the referenced entity (object or part).

## Scoring recipe

```python
ious = []
for pred_mask, gt_mask in zip(predictions, ground_truths):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union > 0:
        ious.append(intersection / union)
return np.mean(ious)
```

## Common pitfalls

- Evaluating only on object-level masks ignores the benchmark's core part-level grounding contribution, leading to inflated but misleading scores.
- Models trained exclusively on object-level data often fail to generalize to part-level references without explicit multi-granularity fine-tuning or fine-grained feature exploration.
- Using bounding box metrics instead of pixel-level IoU misaligns with the segmentation task definition and underestimates localization precision.

## Evidence (verbatim from paper)

> To align with the goal of multi-granularity unification, we use mean Intersection-over-Union (mIoU) as the evaluation metric. Combined with the original object-level annotations, RefCOCOm includes 34k masks and 92k references overall.

## Citation

```bibtex
@misc{liu2025refcocom,
  title={Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities},
  author={Jing Liu et al. (2025)},
  year={2025},
  note={arXiv:2504.01954}
}
```

- arXiv: 2504.01954