Moving Object Segmentation Eval

---
name: moving-object-segmentation-eval
description: Evaluates a model's ability to generate and rank spatio-temporal proposals for moving objects in video. It probes motion-based segmentation quality, proposal coverage, and ranking accuracy on both rigid and non-rigid motion across diverse scenes. Use when the user wants to benchmark on VSB100, Moseg, or asks about evaluating this task. Reports Average best overlap, Coverage.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1412.6504
  bibtex_key: [REDACTED]
  confidence: high
---

# moving-object-segmentation-eval

> Learning to Segment Moving Objects in Videos — Fragkiadaki et al. (2014) (arXiv:1412.6504, 2014)

## What this evaluates

Evaluates a model's ability to generate and rank spatio-temporal proposals for moving objects in video. It probes motion-based segmentation quality, proposal coverage, and ranking accuracy on both rigid and non-rigid motion across diverse scenes.

## Datasets

- **VSB100** — total 100; splits: train (40), test (60)
- **Moseg** — total 59; splits: test (59)

## Metrics

- `Average best overlap` **(primary)** — range: [0, 1]
  - For each ground-truth segment, compute the maximum IoU across all proposals. Average these maximum IoU values across all ground-truth segments.
- `Coverage` **(primary)** — range: [0, 1]
  - Weighted average of IoU scores, where weights correspond to the area of each ground-truth segment. Larger segments contribute more to the score.
- `Detection rate at 50%` — range: percent
  - Percentage of ground-truth segments that achieve a maximum IoU above 0.5 with any proposal.
- `Detection rate at 70%` — range: percent
  - Percentage of ground-truth segments that achieve a maximum IoU above 0.7 with any proposal.
- `Anytime best overlap/detection` — range: [0, 1] | percent
  - Variant of the above metrics where, instead of evaluating per-frame segments, the best overlap/detection is computed across the entire lifespan of each ground-truth tube.

## Input / output format

**Input**: RGB video frames, optical flow fields, and static image segment proposals.

**Output**: Spatio-temporal tubes or per-frame segment proposals, each associated with a bounding box and a moving objectness score.

## Scoring recipe

```python
def compute_metrics(proposals, gt_segments):
    ious = {g: max(iou(p, g) for p in proposals) for g in gt_segments}
    avg_best_overlap = mean(ious.values())
    coverage = sum(ious[g] * area(g) for g in gt_segments) / sum(area(g) for g in gt_segments)
    det_50 = mean(v > 0.5 for v in ious.values())
    det_70 = mean(v > 0.7 for v in ious.values())
    return avg_best_overlap, coverage, det_50, det_70
# For 'anytime best' metrics, replace per-frame segments with spatio-temporal tubes and evaluate across the full lifespan.
```

## Common pitfalls

- Confusing per-frame IoU with spatio-temporal tube IoU, as the paper evaluates both separately and notes different failure modes (e.g., temporal fragmentation vs. background leakage).
- Using mean IoU instead of 'best overlap' or 'coverage', which specifically require taking the maximum IoU per ground-truth object or weighting by ground-truth area.
- Overlooking the 'anytime best' metric, which evaluates proposal quality across an object's entire lifespan rather than frame-by-frame.

## Evidence (verbatim from paper)

> We consider the following four widely used static image segmentation metrics: a) Average best overlap: the average (across all 2D ground-truth segments in our dataset) of the best IoU score of a ground-truth object with all segment proposals. b) Coverage: the weighted average of IoU scores, weighted by the area of the ground-truth segments (larger segments matter more). c) Detection rate at 50%:the percentage of ground-truth segments that have IoU above 50% with a segment proposal. d) Detection rate at 70%.

## Citation

```bibtex
@misc{fragkiadaki2014learning,
  title={Learning to Segment Moving Objects in Videos},
  author={Fragkiadaki et al. (2014)},
  year={2014},
  note={arXiv:1412.6504}
}
```

- arXiv: 1412.6504