Scatspotter Eval

---
name: scatspotter-eval
description: Evaluates object detection and instance segmentation capabilities on real-world images of dog feces. It specifically probes model robustness to camouflage, occlusion, varying lighting conditions, and small object detection in outdoor urban environments. Use when the user wants to benchmark on ScatSpotter, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.16473
  bibtex_key: crall2024scatspotter
  confidence: high
---

# scatspotter-eval

> "ScatSpotter" 2024 -- A Distributed Dog Poop Detection Dataset — Crall (2024) (arXiv:2412.16473, 2024)

## What this evaluates

Evaluates object detection and instance segmentation capabilities on real-world images of dog feces. It specifically probes model robustness to camouflage, occlusion, varying lighting conditions, and small object detection in outdoor urban environments.

## Datasets

- **ScatSpotter** — total 6648; splits: train (5747), val (691), test (30)

## Metrics

- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision computed over IoU thresholds 0.50:0.95 for both bounding box and polygon segmentation tasks, following the standard COCO evaluation protocol.

## Input / output format

**Input**: High-resolution RGB JPEG images (typically 4032x3024) captured in outdoor urban environments.

**Output**: COCO JSON format containing bounding boxes and/or polygon coordinates for each detected instance, along with class labels and confidence scores.

## Scoring recipe

```python
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

def compute_mAP(gt_path, pred_path):
    coco_gt = COCO(gt_path)
    coco_dt = coco_gt.loadRes(pred_path)
    img_ids = sorted(coco_gt.getImgIds())
    eval = COCOeval(coco_gt, coco_dt, 'bbox')
    eval.params.imgIds = img_ids
    eval.evaluate()
    eval.accumulate()
    eval.summarize()
    return eval.stats[0]  # mAP @ IoU 0.50:0.95
```

## Common pitfalls

- Approximately 65% of images are negatives (no annotations) due to the before/after/negative protocol, which can mislead models if treated as positive samples.
- Shadows and low-light conditions frequently cause both annotation errors (SAM failures) and detection failures.
- Small object size and camouflage against natural backgrounds (leaves, dirt, snow) significantly reduce recall.

## Evidence (verbatim from paper)

> Baseline models (ViT, Mask R-CNN) achieve 0.858 mAP on validation, but failure cases highlight persistent difficulties in detecting camouflaged or small poop objects.

## Citation

```bibtex
@misc{crall2024scatspotter,
  title={"ScatSpotter" 2024 -- A Distributed Dog Poop Detection Dataset},
  author={Crall (2024)},
  year={2024},
  note={arXiv:2412.16473}
}
```

- arXiv: 2412.16473