Dronevehicle Eval

---
name: dronevehicle-eval
description: Evaluates aerial vehicle detection capability using aligned RGB and infrared image pairs. It specifically probes a model's ability to fuse cross-modal features and handle uncertainty in low-light or complex urban backgrounds. Use when the user wants to benchmark on DroneVehicle, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.02437
  bibtex_key: sun2020dronevehicle
  confidence: high
---

# dronevehicle-eval

> Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning — Sun et al. (2020) (arXiv:2003.02437, 2020)

## What this evaluates

Evaluates aerial vehicle detection capability using aligned RGB and infrared image pairs. It specifically probes a model's ability to fuse cross-modal features and handle uncertainty in low-light or complex urban backgrounds.

## Datasets

- **DroneVehicle** — total 28439; splits: train (17990), val (1469), test (8980); repo https://github.com/VisDrone/DroneVehicle

## Metrics

- `mAP` **(primary)** — range: percent
  - Mean Average Precision across vehicle categories. A prediction is considered a true positive if the Intersection over Union (IoU) with the nearest ground-truth oriented bounding box exceeds 0.5. AP is computed per class and averaged.

## Input / output format

**Input**: Aligned RGB and infrared aerial image pairs containing vehicles, with ground-truth oriented bounding box annotations for five vehicle categories (car, freight car, truck, bus, van).

**Output**: Oriented bounding boxes (OBB) with class labels and confidence scores for each detected vehicle instance.

## Scoring recipe

```python
def compute_mAP(predictions, ground_truths, iou_thresh=0.5):
    # predictions: list of (class, score, bbox)
    # ground_truths: list of (class, bbox)
    tp, fp = [], []
    for pred in sorted(predictions, key=lambda x: x[1], reverse=True):
        matched = False
        for gt in ground_truths:
            if pred[0] == gt[0] and not gt['used']:
                if compute_iou(pred[2], gt[2]) > iou_thresh:
                    tp.append(1); fp.append(0)
                    gt['used'] = True; matched = True; break
        if not matched: tp.append(0); fp.append(1)
    # Compute precision-recall curve per class, calculate AP, then average
    return average_precision(tp, fp)
```

## Common pitfalls

- Using axis-aligned bounding boxes instead of oriented bounding boxes (OBB), which is required for aerial drone imagery.
- Applying the standard COCO IoU range (0.5:0.95) instead of the fixed 0.5 threshold explicitly stated in the protocol.
- Evaluating single-modality baselines and cross-modal fusion models on different splits or without identical experimental settings.

## Evidence (verbatim from paper)

> The standard metrics, Mean Average Precision (mAP) is adopted to evaluate the drone-based RGB-Infrared vehicle detection accuracy. The mAP measures the quality of bounding box predictions in the test set. Following [29], a prediction is considered as true positive if the IoU between the prediction and its nearest ground-truth annotation is larger than 0.5.

## Citation

```bibtex
@misc{sun2020dronevehicle,
  title={Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning},
  author={Sun et al. (2020)},
  year={2020},
  note={arXiv:2003.02437}
}
```

- arXiv: 2003.02437