Synthref Refvos Eval

---
name: synthref-refvos-eval
description: Evaluates the effectiveness of a synthetic referring expression dataset for training language-guided video object segmentation models. It measures segmentation accuracy when models are trained on synthetic versus human annotations and evaluated on standard referring video segmentation benchmarks. Use when the user wants to benchmark on DAVIS-2017, Refer-YouTube-VOS, or asks about evaluating this task. Reports J&F.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.04403
  bibtex_key: kazakos2021synthref
  confidence: high
---

# synthref-refvos-eval

> SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation — Kazakos et al. (2021) (arXiv:2106.04403, 2021)

## What this evaluates

Evaluates the effectiveness of a synthetic referring expression dataset for training language-guided video object segmentation models. It measures segmentation accuracy when models are trained on synthetic versus human annotations and evaluated on standard referring video segmentation benchmarks.

## Datasets

- **DAVIS-2017** — total ?; splits: val (30), train+val (90)
- **Refer-YouTube-VOS** — total ?; splits: test (-1)

## Metrics

- `J&F` **(primary)** — range: percent
  - Average of Jaccard (J) and F-score (F) indices measuring overlap between predicted and ground-truth segmentation masks. Defined in supplementary material.
- `Prec@0.5` — range: percent
  - Percentage of frames where the Intersection over Union (IoU) between predicted and ground-truth masks exceeds 0.5.
- `Prec@0.9` — range: percent
  - Percentage of frames where the IoU between predicted and ground-truth masks exceeds 0.9.
- `Mean IoU` — range: percent
  - Average Intersection over Union across all frames and objects in the evaluation set.

## Input / output format

**Input**: Video sequences paired with referring expressions (text) and ground-truth segmentation masks.

**Output**: Per-frame binary segmentation masks for the referred object.

## Scoring recipe

```python
def compute_metrics(pred_masks, gt_masks):
    ious = [iou(p, g) for p, g in zip(pred_masks, gt_masks)]
    j = mean(ious)
    f = f_score(pred_masks, gt_masks)
    jf = (j + f) / 2
    prec_05 = mean([1 if i > 0.5 else 0 for i in ious]) * 100
    prec_09 = mean([1 if i > 0.9 else 0 for i in ious]) * 100
    mean_iou = mean(ious) * 100
    return jf, prec_05, prec_09, mean_iou
```

## Common pitfalls

- J&F metric definition is located in the supplementary material, not the main text.
- Refer-YouTube-VOS evaluation uses only a subset of videos that correspond to the synthetic dataset, not the full benchmark.
- Performance gains vary significantly depending on whether the model is evaluated after zero-shot pretraining or after fine-tuning on target domain data.

## Evidence (verbatim from paper)

> DAVIS-2017 We report the gains of adding the synthetic dataset when evaluating on the standard validation partition of DAVIS-2017 (30 videos), but also on the combined training and validation partitions (90 videos), to obtain more statistically significant results. The J&F metric is defined in the supplementary material.

## Citation

```bibtex
@misc{kazakos2021synthref,
  title={SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation},
  author={Kazakos et al. (2021)},
  year={2021},
  note={arXiv:2106.04403}
}
```

- arXiv: 2106.04403