Florence 2 Eval

---
name: florence-2-eval
description: Evaluates a unified vision foundation model's zero-shot and fine-tuned capabilities across diverse computer vision tasks. It probes the model's ability to perform image captioning, visual question answering, object detection, referring expression comprehension, and semantic segmentation using a single sequence-to-sequence architecture. Use when the user wants to benchmark on COCO, Flickr30k, RefCOCO/+/g, VQAv2, ADE20K, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.06242
  bibtex_key: xiao2023florence2
  confidence: high
---

# florence-2-eval

> Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks — Bin Xiao et al. (arXiv:2311.06242, 2023)

## What this evaluates

Evaluates a unified vision foundation model's zero-shot and fine-tuned capabilities across diverse computer vision tasks. It probes the model's ability to perform image captioning, visual question answering, object detection, referring expression comprehension, and semantic segmentation using a single sequence-to-sequence architecture.

## Datasets

- **COCO** — total ?; splits: train2017 (-1), val2017 (-1), Karpathy test (-1)
- **Flickr30k** — total ?; splits: test (-1)
- **RefCOCO/+/g** — total ?; splits: val (-1), test-A (-1), test-B (-1)
- **VQAv2** — total ?; splits: test-dev (-1)
- **ADE20K** — total ?; splits: val (-1)

## Metrics

- `CIDEr` **(primary)** — range: [0, inf)
  - Consensus-based Image Description Evaluation. Computes n-gram overlap between generated captions and multiple ground-truth captions, weighted by inverse document frequency (IDF) scores.
- `mAP` — range: [0, 1]
  - Mean Average Precision at IoU thresholds 0.5:0.95 (APb for detection, APm for segmentation).
- `Recall@1` — range: [0, 1]
  - Percentage of queries where the correct bounding box is ranked first among predictions.
- `Accuracy` — range: [0, 1]
  - Exact match ratio between predicted class/label and ground truth for VQA and referring expression comprehension tasks.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union between predicted segmentation masks and ground truth masks across all classes.

## Input / output format

**Input**: RGB image resized to 384x384 (or 768x768 for high-res tuning) concatenated with a task-specific text prompt (e.g., '<CAPTION>', '<DETECT>', '<REFERRING_EXPRESSION>') in a sequence-to-sequence format.

**Output**: Text sequence generated by the decoder, representing a caption, bounding box coordinates (normalized), segmentation mask tokens, or a class label.

## Scoring recipe

```python
def score(predictions, golds, task):
    if task == 'captioning': return cider_score(predictions, golds)
    if task == 'detection': return map_score(predictions, golds, iou=0.5)
    if task == 'referring': return recall_at_1(predictions, golds)
    if task == 'vqa': return accuracy(predictions, golds)
    if task == 'segmentation': return miou_score(predictions, golds)
    return 0
```

## Common pitfalls

- Using the standard COCO test set instead of the Karpathy split for captioning evaluation, which yields different scores.
- Comparing VQA accuracy without accounting for external OCR usage in baseline models, as some baselines use OCR tokens while Florence-2 does not.
- Confusing zero-shot evaluation (no task-specific fine-tuning) with generalist fine-tuning (single model trained on multiple public datasets).

## Evidence (verbatim from paper)

> For image-level tasks, Florence-2-L achieves a 135.6 CIDEr score on the COCO caption benchmark... For region-level grounding and referring expression comprehension tasks, Florence-2-L establishes a new record in zero-shot performance achieving a 5.7 improvement in Flickr30k Recall@1... Additionally, our pre-trained model attains a 35.8% mIOU in the Refcoco referring expression segmentation (RES) task

## Citation

```bibtex
@misc{xiao2023florence2,
  title={Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks},
  author={Bin Xiao et al.},
  year={2023},
  note={arXiv:2311.06242}
}
```

- arXiv: 2311.06242