Roboflow 100 Eval

---
name: roboflow-100-eval
description: Evaluates object detection models' ability to generalize across diverse, real-world, domain-specific visual tasks. It probes fine-tuning performance and zero-shot transfer capabilities on crowdsourced, practitioner-curated datasets spanning multiple imaging modalities. Use when the user wants to benchmark on Roboflow 100, or asks about evaluating this task. Reports mAP@.50.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.13523
  bibtex_key: ciaglia2022roboflow100
  confidence: high
---

# roboflow-100-eval

> Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark — Ciaglia et al. (2022) (arXiv:2211.13523, 2022)

## What this evaluates

Evaluates object detection models' ability to generalize across diverse, real-world, domain-specific visual tasks. It probes fine-tuning performance and zero-shot transfer capabilities on crowdsourced, practitioner-curated datasets spanning multiple imaging modalities.

## Datasets

- **Roboflow 100** — total 224714; splits: train (-1), val (-1), test (-1)

## Metrics

- `mAP@.50` **(primary)** — range: [0, 1]
  - Mean Average Precision at Intersection over Union (IoU) threshold of 0.50. Computed per class and averaged across all classes in a dataset or category.
- `mAP@.50:.95` — range: [0, 1]
  - Mean Average Precision averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05. Used primarily for the GLIP zero-shot model evaluation.

## Input / output format

**Input**: RGB images resized to 640x640 pixels with corresponding bounding box annotations and class labels.

**Output**: Predicted bounding boxes with class labels and confidence scores.

## Scoring recipe

```python
def compute_map(predictions, ground_truth, iou_thresh=0.5):
    # 1. Group predictions and ground truth by class
    # 2. For each class, sort predictions by confidence score descending
    # 3. Match each prediction to the highest IoU ground truth box (IoU >= iou_thresh)
    # 4. Mark matches as True Positives, unmatched as False Positives
    # 5. Compute precision and recall at each threshold, build PR curve
    # 6. Interpolate AP at 11 recall points (or use modern trapezoidal rule)
    # 7. Average AP across all classes to yield mAP@.50
    return mAP_value
```

## Common pitfalls

- Split ratios are not standardized across the 100 datasets; they follow original authors' splits unless underrepresented, complicating cross-dataset comparison.
- Class names vary in descriptiveness, which heavily impacts zero-shot models like GLIP that rely on text prompts for detection.
- Bounding box sizes differ drastically across domains (e.g., aerial vs. documents), affecting detection difficulty and metric interpretation.

## Evidence (verbatim from paper)

> We trained both models with default hyperparameters for 100 epochs at 640x640 resolution. We recorded the average mAP@.50 value for the YOLOv5 and YOLOv7 models and the mAP@.50:.95 for the GLIP model for each category.

## Citation

```bibtex
@misc{ciaglia2022roboflow100,
  title={Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
  author={Ciaglia et al. (2022)},
  year={2022},
  note={arXiv:2211.13523}
}
```

- arXiv: 2211.13523