Vos Language Referring

---
name: vos-language-referring
description: Evaluates a model's ability to perform pixel-level video object segmentation guided by natural language referring expressions, testing both language grounding and temporal consistency in dynamic scenes. Use when the user wants to benchmark on DAVIS-16, DAVIS-17, or asks about evaluating this task. Reports performance score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.08006
  bibtex_key: [REDACTED]
  confidence: medium
---

# vos-language-referring

> Video Object Segmentation with Language Referring Expressions — Khoreva et al. (2018) (arXiv:1803.08006, 2018)

## What this evaluates

Evaluates a model's ability to perform pixel-level video object segmentation guided by natural language referring expressions, testing both language grounding and temporal consistency in dynamic scenes.

## Datasets

- **DAVIS-16** — total ?; splits: train (-1), val (-1), test (-1)
- **DAVIS-17** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `performance score` **(primary)** — range: percent
  - Not explicitly defined in the provided text; standard video object segmentation metrics (e.g., J&F or mIoU) are implied for comparing against mask-based and scribble-based baselines.

## Input / output format

**Input**: A video sequence V={f_1,...,f_N} and a natural language referring expression Q describing the target object.

**Output**: A pixel-level segmentation mask for the target object in every frame where it appears.

## Scoring recipe

```python
# Not explicitly provided in text.
# Standard VOS evaluation typically follows:
for frame in video:
    pred_mask = model.predict(frame, query)
    gt_mask = get_ground_truth(frame)
    iou = intersection(pred_mask, gt_mask) / union(pred_mask, gt_mask)
    scores.append(iou)
final_score = mean(scores)
```

## Common pitfalls

- Using image-only grounding models per frame leads to temporally inconsistent and jittery bounding box predictions.
- Training on static images (MSRA) and testing on video can cause domain shift, requiring careful augmentation (e.g., synthesized optical flow, box jittering).

## Evidence (verbatim from paper)

> demonstrating that language supervision achieves performance comparable to mask-based methods on DAVIS-16 and matches scribble-based approaches on the challenging DAVIS-17 dataset

## Citation

```bibtex
@misc{khoreva2018voslanguage,
  title={Video Object Segmentation with Language Referring Expressions},
  author={Khoreva et al. (2018)},
  year={2018},
  note={arXiv:1803.08006}
}
```

- arXiv: 1803.08006