Yoloe Lvis Coco Eval

---
name: yoloe-lvis-coco-eval
description: Evaluates open-vocabulary object detection and segmentation capabilities using text, visual, and prompt-free inputs on zero-shot and fine-tuned settings. Use when the user wants to benchmark on LVIS, COCO, or asks about evaluating this task. Reports Fixed AP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07465
  bibtex_key: wang2025yoloe
  confidence: high
---

# yoloe-lvis-coco-eval

> YOLOE: Real-Time Seeing Anything — Wang et al. (2025) (arXiv:2503.07465, 2025)

## What this evaluates

Evaluates open-vocabulary object detection and segmentation capabilities using text, visual, and prompt-free inputs on zero-shot and fine-tuned settings.

## Datasets

- **LVIS** — total ?; splits: minival (-1), val (-1)
- **COCO** — total ?; splits: train (-1), val (-1)

## Metrics

- `Fixed AP` **(primary)** — range: [0, 100]
  - Average precision computed with a fixed number of detections per image (typically 100), evaluated in a zero-shot manner on the LVIS minival subset.
- `APm` — range: [0, 100]
  - Mean average precision for segmentation masks, evaluated on the LVIS val set.
- `AP` — range: [0, 100]
  - Standard average precision for detection and segmentation, evaluated on COCO after fine-tuning.
- `FPS` — range: other
  - Frames per second measured on Nvidia T4 GPU with TensorRT and iPhone 12 with CoreML.

## Input / output format

**Input**: RGB image; optional text prompt (category names), visual prompt (bounding box crops), or no prompt (prompt-free).

**Output**: List of predicted bounding boxes and/or segmentation masks with associated confidence scores and category labels.

## Scoring recipe

```python
def compute_fixed_ap(predictions, ground_truth):
    # predictions: list of dicts with 'bbox', 'score', 'category'
    # ground_truth: list of dicts with 'bbox', 'category'
    # 1. Sort predictions by score descending.
    # 2. Keep top 100 detections per image.
    # 3. Match to ground truth using IoU >= 0.5.
    # 4. Compute precision-recall curve and average.
    return ap_score
```

## Common pitfalls

- Fixed AP on LVIS minival uses a fixed number of detections per image, unlike standard AP which uses all detections.
- Prompt-free evaluation maps open-ended predictions to a built-in vocabulary of 4585 categories using a text encoder, rather than the LVIS category list.
- Visual prompt evaluation averages embeddings from N=16 randomly sampled training images per category, not test images.

## Evidence (verbatim from paper)

> By default, Fixed AP [[7]] on LVIS minival subset is reported. For transferring to COCO, standard AP is evaluated, following [[1], [21]].

## Citation

```bibtex
@misc{wang2025yoloe,
  title={YOLOE: Real-Time Seeing Anything},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.07465}
}
```

- arXiv: 2503.07465