Video Class Agnostic Eval

---
name: video-class-agnostic-eval
description: Evaluates a model's ability to segment moving and unknown objects in autonomous driving videos without relying on a closed set of known classes. It probes open-set and motion-based instance segmentation capabilities under varying data distributions and synthetic scenarios. Use when the user wants to benchmark on Cityscapes-VPS, KITTI-MOTS, Carla, or asks about evaluating this task. Reports CAQ, CA-IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.11015
  bibtex_key: siam2021video
  confidence: high
---

# video-class-agnostic-eval

> Video Class Agnostic Segmentation Benchmark for Autonomous Driving — Siam et al. (2021) (arXiv:2103.11015, 2021)

## What this evaluates

Evaluates a model's ability to segment moving and unknown objects in autonomous driving videos without relying on a closed set of known classes. It probes open-set and motion-based instance segmentation capabilities under varying data distributions and synthetic scenarios.

## Datasets

- **Cityscapes-VPS** — total ?; splits: train (-1), test (-1)
- **KITTI-MOTS** — total ?; splits: train (-1), test (-1)
- **Carla** — total ?; splits: train (-1), test (-1)

## Metrics

- `CAQ` **(primary)** — range: percent
  - Class Agnostic Quality, measuring the segmentation quality of moving or unknown objects independent of their semantic class.
- `CA-IoU` **(primary)** — range: percent
  - Intersection over Union computed exclusively on pixels belonging to unknown or class-agnostic objects during testing.
- `mIoU` — range: percent
  - Mean Intersection over Union across all predicted semantic classes.
- `PQ` — range: percent
  - Panoptic Quality, combining segmentation quality (SQ) and recognition quality (RQ) for all, thresholded, and static instances.

## Input / output format

**Input**: Monocular video frames (typically resized to 1024x512) optionally accompanied by depth maps (simulated ground truth or estimated).

**Output**: Per-pixel segmentation masks for known semantic classes, moving instances, and a unified class-agnostic mask for unknown objects.

## Scoring recipe

```python
def compute_ca_iou(pred_mask, gt_mask, unknown_class_ids):
    pred_unk = np.isin(pred_mask, unknown_class_ids)
    gt_unk = np.isin(gt_mask, unknown_class_ids)
    intersection = np.logical_and(pred_unk, gt_unk).sum()
    union = np.logical_or(pred_unk, gt_unk).sum()
    return (intersection / union) * 100 if union > 0 else 0.0
```

## Common pitfalls

- Withholding specific classes (Person, Rider, Motorcycle, Bicycle) as 'unknown' during training but only evaluating CA-IoU on a subset (e.g., Motorcycle and Bicycle) during testing.
- Using different sets of unknown objects for training and testing to prevent overfitting, which complicates direct comparison across papers.
- Depth modality is not standardized: synthetic data uses ground-truth depth while real data uses estimated depth, affecting reproducibility.

## Evidence (verbatim from paper)

> The CA-IoU is reported only on the unknown objects used during testing, which are labelled in Figure 4.

## Citation

```bibtex
@misc{siam2021video,
  title={Video Class Agnostic Segmentation Benchmark for Autonomous Driving},
  author={Siam et al. (2021)},
  year={2021},
  note={arXiv:2103.11015}
}
```

- arXiv: 2103.11015