Mosev2 Eval

---
name: mosev2-eval
description: This benchmark evaluates video object segmentation and tracking models under highly complex, unconstrained real-world conditions. It specifically probes robustness to severe occlusions, frequent object disappearance and reappearance, adverse weather, low-light environments, camouflage, and knowledge-dependent scenarios. Use when the user wants to benchmark on MOSEv2, or asks about evaluating this task. Reports $\mathcal{J}\\&\dot{\mathcal{F}}$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05630
  bibtex_key: ding2025mosev2
  confidence: high
---

# mosev2-eval

> MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes — Ding et al. (2025) (arXiv:2508.05630, 2025)

## What this evaluates

This benchmark evaluates video object segmentation and tracking models under highly complex, unconstrained real-world conditions. It specifically probes robustness to severe occlusions, frequent object disappearance and reappearance, adverse weather, low-light environments, camouflage, and knowledge-dependent scenarios.

## Datasets

- **MOSEv2** — total 4713; splits: train (3666), val (433), test (614)

## Metrics

- `$\mathcal{J}\\&\dot{\mathcal{F}}$` **(primary)** — range: percent
  - The mean of the Jaccard index ($\mathcal{J}$) and the adaptive F-score ($\dot{\mathcal{F}}$). The adaptive F-score uses a dynamic threshold to better assess boundary quality across objects of varying sizes, yielding stricter scores than the standard F-score.
- `$\mathcal{J}\\&\dot{\mathcal{F}}_{d}$` — range: percent
  - J&F dot computed only on frames where the target object is disappearing or occluded.
- `$\mathcal{J}\\&\dot{\mathcal{F}}_{r}$` — range: percent
  - J&F dot computed only on frames where the target object reappears after being absent.

## Input / output format

**Input**: For semi-supervised VOS: a video sequence plus an initial prompt on the first frame (full mask, bounding box, or point coordinates). For unsupervised VOS: video sequence only (with a proposal limit of 20). For interactive VOS: video sequence plus iterative user clicks.

**Output**: Per-frame binary segmentation masks (or instance masks) for the target object(s) across the entire video sequence.

## Scoring recipe

```python
def compute_jf_dot(pred_masks, gt_masks):
    j_scores = [jaccard(p, g) for p, g in zip(pred_masks, gt_masks)]
    f_scores = [adaptive_fscore(p, g) for p, g in zip(pred_masks, gt_masks)]
    jf_scores = [(j + f) / 2 for j, f in zip(j_scores, f_scores)]
    return sum(jf_scores) / len(jf_scores)
```

## Common pitfalls

- Using the standard F-score ($\mathcal{F}$) instead of the adaptive F-score ($\dot{\mathcal{F}}$), which overestimates boundary accuracy for small or irregular objects.
- Failing to limit unsupervised methods to a maximum of 20 object proposals per video, leading to unfair performance inflation.
- Confusing disappearance ($\mathcal{J}\\&\dot{\mathcal{F}}_{d}$) and reappearance ($\mathcal{J}\\&\dot{\mathcal{F}}_{r}$) sub-metrics, which require strict temporal masking of frames where the target is absent vs. present.

## Evidence (verbatim from paper)

> We evaluate model performance using standard metrics ($\mathcal{J}$, $\mathcal{F}$, and $\mathcal{J}\\&\mathcal{F}$) on MOSEv2 validation set, following the DAVIS protocol*[[2], [3]]*. To better capture the complex challenges in MOSEv2, we additionally report $\dot{\mathcal{F}}$, $\mathcal{J}\\&\dot{\mathcal{F}}$, $\mathcal{J}\\&\dot{\mathcal{F}}_{d}$, and $\mathcal{J}\\&\dot{\mathcal{F}}_{r}$ as described in [Section 3.3]. Among them, $\mathcal{J}\\&\dot{\mathcal{F}}$ is selected as the primary evaluation metric.

## Citation

```bibtex
@misc{ding2025mosev2,
  title={MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes},
  author={Ding et al. (2025)},
  year={2025},
  note={arXiv:2508.05630}
}
```

- arXiv: 2508.05630