Segmentmeifyoucan Eval

---
name: segmentmeifyoucan-eval
description: This benchmark evaluates a model's ability to segment anomalous or hazardous objects in driving scenes that were not seen during training. It probes out-of-distribution detection and pixel-wise localization of unknown road obstacles, emphasizing safety-critical detection regardless of object class. Use when the user wants to benchmark on RoadAnomaly21, RoadObstacle21, or asks about evaluating this task. Reports AuPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.14812
  bibtex_key: chan2021segmentmeifyoucan
  confidence: high
---

# segmentmeifyoucan-eval

> SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation — Chan et al. (2021) (arXiv:2104.14812, 2021)

## What this evaluates

This benchmark evaluates a model's ability to segment anomalous or hazardous objects in driving scenes that were not seen during training. It probes out-of-distribution detection and pixel-wise localization of unknown road obstacles, emphasizing safety-critical detection regardless of object class.

## Datasets

- **RoadAnomaly21** — total 100; splits: test (100); repo https://github.com/SegmentMeIfYouCan/road-anomaly-benchmark
- **RoadObstacle21** — total 327; splits: test (327); repo https://github.com/SegmentMeIfYouCan/road-anomaly-benchmark

## Metrics

- `AuPRC` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed over pixel-wise anomaly scores. Emphasizes detecting the minority anomaly class in highly unbalanced pixel distributions.
- `FPR95` — range: percent
  - False positive rate when the true positive rate reaches 95%. Indicates how many false positives must be made to achieve the desired safety threshold.
- `F1` — range: [0, 1]
  - Component-wise F1-score: F1(τ) = 2·TP(τ) / (2·TP(τ) + FN(τ) + FP(τ)). Summarizes true positives, false negatives, and false positives at a given threshold τ.

## Input / output format

**Input**: RGB image of a driving scene. The model receives the image and outputs pixel-wise anomaly scores or a binary segmentation mask.

**Output**: Pixel-wise anomaly scores s(x) ∈ ℝ^|Z| or a binary mask. Predictions are thresholded at τ ∈ [0,1) to form connected components for component-level evaluation.

## Scoring recipe

```python
def compute_metrics(gold_mask, pred_scores, tau=0.5):
    # Pixel-level
    au_prc = auc_precision_recall(gold_mask, pred_scores)
    fpr95 = fpr_at_tpr(gold_mask, pred_scores, target_tpr=0.95)

    # Component-level
    gt_comps = find_connected_components(gold_mask, class='anomaly')
    pred_comps = find_connected_components(pred_scores > tau, class='anomaly')
    tp, fn, fp = 0, 0, 0
    for k in gt_comps:
        union_k = union(p for p in pred_comps if p & k)
        sIoU_k = len(k & union_k) / len((k | union_k) - A(k))
        tp += 1 if sIoU_k > tau else 0
        fn += 1 if sIoU_k <= tau else 0
    for k_hat in pred_comps:
        ppv_k = len(k_hat & union(g for g in gt_comps if g & k_hat)) / len(k_hat)
        fp += 1 if ppv_k <= tau else 0
    f1 = 2 * tp / (2 * tp + fn + fp)
    return au_prc, fpr95, f1
```

## Common pitfalls

- Void regions must be strictly excluded from evaluation; predictions falling in void areas are not counted as false positives.
- Component-level metrics use adjusted IoU (sIoU) that excludes pixels correctly intersecting other ground-truth components, unlike standard IoU which penalizes covering multiple objects.
- Threshold τ significantly impacts component-level TP/FN/FP counts; results must be reported at specific thresholds (0.25, 0.50, 0.75) or averaged, not assumed.

## Evidence (verbatim from paper)

> We evaluate the separability of the pixel-wise anomaly scores via the area under the precision-recall curve (AuPRC)... To consider the safety point of view, we also include the false positive rate at 95% true positive rate (FPR95) in our evaluation... As an overall metric, we additionally include the component-wise F1-score defined as F1(τ) = 2·TP(τ) / (2·TP(τ) + FN(τ) + FP(τ)) ∈ [0,1], which summarizes the TP, FN and FP quantities (that depend on τ).

## Citation

```bibtex
@misc{chan2021segmentmeifyoucan,
  title={SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation},
  author={Chan et al. (2021)},
  year={2021},
  note={arXiv:2104.14812}
}
```

- arXiv: 2104.14812