Road Anomaly Seg Eval

---
name: road-anomaly-seg-eval
description: Evaluates video-level road anomaly segmentation models in autonomous driving scenarios, specifically probing their ability to maintain prediction validity over time sequences and perform under real-time latency constraints. Use when the user wants to benchmark on Road Anomaly Segmentation Dataset, or asks about evaluating this task. Reports latency-aware metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.04942
  bibtex_key: tian2024latency
  confidence: medium
---

# road-anomaly-seg-eval

> Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics — Tian et al. (2024) (arXiv:2401.04942, 2024)

## What this evaluates

Evaluates video-level road anomaly segmentation models in autonomous driving scenarios, specifically probing their ability to maintain prediction validity over time sequences and perform under real-time latency constraints.

## Datasets

- **Road Anomaly Segmentation Dataset** — total 220; splits: train (200), val (20)

## Metrics

- `latency-aware metrics` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic (AUROC) and False Positive Rate at 95% True Positive Rate (FPR95) computed over video sequences, adjusted to account for inference latency constraints during streaming evaluation.
- `temporal consistency` — range: [0, 1]
  - Measures the stability of anomaly predictions across consecutive frames in a video sequence, penalizing flickering or inconsistent segmentations over time.

## Input / output format

**Input**: Video sequences of road scenes (synthetic CARLA data) with ground-truth anomaly segmentation masks.

**Output**: Per-frame anomaly segmentation masks for each video sequence.

## Scoring recipe

```python
def evaluate(predictions, gold, timestamps, latency_budget):
    valid_preds = [p for p, t in zip(predictions, timestamps) if t <= latency_budget]
    valid_gold = [g for g, t in zip(gold, timestamps) if t <= latency_budget]
    frame_scores = [compute_iou(p, g) for p, g in zip(valid_preds, valid_gold)]
    auroc = compute_auroc(frame_scores, valid_gold)
    fpr95 = compute_fpr_at_tpr95(frame_scores, valid_gold)
    consistency = compute_frame_to_frame_iu_correlation(predictions)
    return auroc, fpr95, consistency
```

## Common pitfalls

- Evaluating only on latency-agnostic metrics ignores real-world streaming constraints, overestimating practical utility.
- Retraining models on extra anomalous data often improves latency-agnostic scores but degrades temporal consistency due to overfitting to specific anomaly appearances.
- High latency-agnostic performance correlates with higher sensitivity to latency drops, making latency-aware evaluation essential for safety-critical deployment.

## Evidence (verbatim from paper)

> During evaluation, we utilize an additional set of scenes without anomalous objects for the training of the semantic segmentation task, which is a preceding task of anomaly segmentation. Of the 220 video sequences, 200 are used for training (if needed) and 20 are used for validation. The results evaluated on the proposed metrics, i.e., the latency-agnostic, latency-aware metrics as well as the temporal consistency, are reported in Table 1.

## Citation

```bibtex
@misc{tian2024latency,
  title={Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2401.04942}
}
```

- arXiv: 2401.04942