Mcity Data Engine Eval

---
name: mcity-data-engine-eval
description: Evaluates an open-vocabulary data selection pipeline for iteratively improving object detection models on rare classes. Specifically, it tests whether adding newly selected and labeled frames of vulnerable road users (pedestrians, cyclists) to a seed dataset improves detection performance on fisheye traffic camera data. Use when the user wants to benchmark on SIP VRU Detection Dataset, or asks about evaluating this task. Reports mAP@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.21614
  bibtex_key: bogdoll2025mcity
  confidence: high
---

# mcity-data-engine-eval

> Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection — Bogdoll et al. (2025) (arXiv:2504.21614, 2025)

## What this evaluates

Evaluates an open-vocabulary data selection pipeline for iteratively improving object detection models on rare classes. Specifically, it tests whether adding newly selected and labeled frames of vulnerable road users (pedestrians, cyclists) to a seed dataset improves detection performance on fisheye traffic camera data.

## Datasets

- **SIP VRU Detection Dataset** — total ?; splits: seed (1260), iterative (2352), validation (-1); repo https://github.com/mcity/mcity_data_engine

## Metrics

- `mAP@0.5` **(primary)** — range: [0, 1]
  - Mean Average Precision at Intersection over Union (IoU) threshold of 0.5. Computed by averaging the precision-recall curve across all confidence thresholds for the target classes (pedestrian, cyclist).
- `F1 Score` — range: [0, 1]
  - Harmonic mean of precision and recall calculated at a single confidence threshold, representing the balance between false positives and false negatives for deployment settings.

## Input / output format

**Input**: Fisheye camera frames from the Smart Intersections Project traffic stream.

**Output**: Bounding box predictions with class labels (pedestrian, cyclist) and confidence scores. The evaluation enforces a 3/5 majority consensus across an ensemble of five open-vocabulary models before accepting a detection.

## Scoring recipe

```python
# Ensemble prediction with majority voting
preds = [model.predict(frame) for model in ensemble]
final_preds = filter_consensus(preds, min_votes=3)
# Match against ground truth
tp, fp, fn = match(final_preds, gt, iou_thresh=0.5)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall)
# mAP@0.5 across thresholds
map50 = compute_map(tp_scores, iou_thresh=0.5)
```

## Common pitfalls

- The ensemble requires a strict 3/5 majority consensus; ignoring this leads to inflated false positive rates and invalidates the reported metrics.
- Crowd VRUs (>40 per frame) are treated as a separate task and excluded from the primary iterative dataset ($\mathcal{D}_{iter}$), so mixing them in changes the evaluation scope.
- mAP@0.5 is computed over multiple confidence thresholds, while F1/Precision/Recall are evaluated at a single deployment threshold; comparing them directly without noting the threshold difference is misleading.

## Evidence (verbatim from paper)

> While mAP@0.5 is the most comprehensive metric for overall model performance over multiple confidence thresholds, the remaining metrics evaluate the model performance based on a single confidence threshold, which is closer to a deployment setting. The F1 score represents the balance between recall and precision. High recall reduces false negatives, while high precision reduces false positives.

## Citation

```bibtex
@misc{bogdoll2025mcity,
  title={Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection},
  author={Bogdoll et al. (2025)},
  year={2025},
  note={arXiv:2504.21614}
}
```

- arXiv: 2504.21614