Contextual Object Detection Eval

---
name: contextual-object-detection-eval
description: Probes a multimodal large language model's ability to infer and localize objects within human-AI interaction contexts (e.g., cloze tests, captioning, QA) using open-vocabulary object names, rather than fixed class sets. Use when the user wants to benchmark on CODE, or asks about evaluating this task. Reports Acc@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.18279
  bibtex_key: zang2023contextdet
  confidence: high
---

# contextual-object-detection-eval

> Contextual Object Detection with Multimodal Large Language Models — Zang et al. (2023) (arXiv:2305.18279, 2023)

## What this evaluates

Probes a multimodal large language model's ability to infer and localize objects within human-AI interaction contexts (e.g., cloze tests, captioning, QA) using open-vocabulary object names, rather than fixed class sets.

## Datasets

- **CODE** — total 708863; splits: train (665161), val (22061), test (21641); repo https://github.com/yuhangzang/ContextDET

## Metrics

- `Acc@1` **(primary)** — range: percent
  - Percentage of correctly predicted object words in the cloze test setting, where the model generates a single top-1 word to fill a masked token in a caption.
- `Acc@5` — range: percent
  - Percentage of cases where the ground-truth object word appears in the model's top-5 predicted words, used to account for synonym variation.
- `AP@1` — range: percent
  - Mean Average Precision computed over bounding boxes predicted using the model's top-1 object word.
- `AP@5` — range: percent
  - Mean Average Precision computed over bounding boxes predicted using the model's top-5 object words.

## Input / output format

**Input**: An image, a caption with a masked object name (e.g., 'A [MASK] is sitting on the grass'), and optionally a question or context for QA/captioning settings.

**Output**: A predicted object word (top-1 or top-5) and corresponding bounding box coordinates for the detected object.

## Scoring recipe

```python
def score(predictions, golds):
    # Text accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if g in p[:5])
    acc = (correct / len(golds)) * 100
    
    # Box AP (standard COCO-style mAP with IoU=0.5)
    # AP@1 uses boxes paired with top-1 predictions
    # AP@5 uses boxes paired with top-5 predictions
    ap1 = compute_mAP(predictions_boxes_top1, gold_boxes, iou_thresh=0.5)
    ap5 = compute_mAP(predictions_boxes_top5, gold_boxes, iou_thresh=0.5)
    return acc, ap1, ap5
```

## Common pitfalls

- Evaluating on fixed class IDs instead of flexible object name words, which defeats the open-vocabulary purpose.
- Ignoring top-5 metrics (Acc@5, AP@5), which are crucial due to synonym variation and fine-grained vocabulary challenges.
- Assuming standard closed-set detection; the task requires generating language first before detecting boxes.

## Evidence (verbatim from paper)

> In our contextual cloze test setting, we compute accuracy by calculating the percentage of correctly predicted object words. However, evaluating this accuracy poses a challenge due to the presence of numerous synonyms and fine-grained object words in human language, which can be difficult for annotators to distinguish. This is a problem similar to those faced by previous large vocabulary image-classification datasets, such as ImageNet, which use the top-5 accuracy metric as a supplementary metric to the top-1 accuracy. Consequently, we also adopt both the top-1 accuracy (Acc@1) and the top-5 accuracy (Acc@5) as our evaluation metrics. For box evaluation, we compute the mean Average Precision (mAP) metric based on the top-1 and top-5 predicted names, which are represented as AP@1 and AP@5.

## Citation

```bibtex
@misc{zang2023contextdet,
  title={Contextual Object Detection with Multimodal Large Language Models},
  author={Zang et al. (2023)},
  year={2023},
  note={arXiv:2305.18279}
}
```

- arXiv: 2305.18279