V3det Eval

---
name: v3det-eval
description: Probes an object detector's ability to localize and classify instances across a vast, hierarchical vocabulary of over 13,000 categories. It evaluates both closed-set detection performance and open-vocabulary generalization to novel, unseen categories. Use when the user wants to benchmark on V3Det, or asks about evaluating this task. Reports AP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.03752
  bibtex_key: wang2023v3det
  confidence: high
---

# v3det-eval

> V3Det: Vast Vocabulary Visual Detection Dataset — Jiaqi Wang et al. (2023) (arXiv:2304.03752, 2023)

## What this evaluates

Probes an object detector's ability to localize and classify instances across a vast, hierarchical vocabulary of over 13,000 categories. It evaluates both closed-set detection performance and open-vocabulary generalization to novel, unseen categories.

## Datasets

- **V3Det** — total 243038; splits: train (183354), val (29821), test (29863)

## Metrics

- `AP` **(primary)** — range: [0, 1]
  - Mean Average Precision computed across IoU thresholds from 0.5 to 0.95, following the COCO evaluation protocol. It averages the AP at each IoU threshold.

## Input / output format

**Input**: RGB image with associated ground-truth bounding boxes and category labels. For open-vocabulary evaluation, text descriptions of base and novel categories are also provided.

**Output**: List of predicted bounding boxes, each containing a class label, confidence score, and bounding box coordinates (x, y, width, height).

## Scoring recipe

```python
def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    predictions = sorted(predictions, key=lambda x: x['score'], reverse=True)
    tp, fp = [], []
    for pred in predictions:
        gt_match = find_best_match(pred, ground_truth, iou_thresh)
        if gt_match and not gt_match['used']:
            tp.append(1); gt_match['used'] = True
        else:
            fp.append(1)
    tp_cum = np.cumsum(tp)
    fp_cum = np.cumsum(fp)
    precision = tp_cum / (tp_cum + fp_cum + 1e-8)
    recall = tp_cum / (len(ground_truth) + 1e-8)
    return compute_ap_from_curve(precision, recall)
# Final AP is mean of AP across IoU thresholds 0.50:0.05:0.95
```

## Common pitfalls

- The test split images are not publicly released; researchers must upload predictions to a designated public test server for evaluation.
- Open-vocabulary benchmarks require strict separation of base ($C_{base}$) and novel ($C_{novel}$) classes, reporting $bAP$ and $nAP$ separately rather than a single aggregate score.
- The dataset contains 13,204 categories with hierarchical inclusion relationships, so standard flat-category detectors may struggle without specific adaptations like the Norm Linear Layer or repeat factor sampler.

## Evidence (verbatim from paper)

> We follow the evaluation metrics of the COCO dataset to report the mean average precise (AP) on different IoU thresholds (i.e., 0.5 ~ 0.95).

## Citation

```bibtex
@misc{wang2023v3det,
  title={V3Det: Vast Vocabulary Visual Detection Dataset},
  author={Jiaqi Wang et al. (2023)},
  year={2023},
  note={arXiv:2304.03752}
}
```

- arXiv: 2304.03752