Visual Prompt Eval

---
name: visual-prompt-eval
description: Evaluates multimodal large language models' ability to comprehend and reason about visual prompts (points, bounding boxes, free-form shapes) for fine-grained object classification, region captioning, OCR, and complex visual reasoning. Use when the user wants to benchmark on LVIS, PACO, COCO-Text, RefCOCOg, MDVP-Bench, LLaVA-Bench, Ferret-Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.20271
  bibtex_key: [REDACTED]
  confidence: high
---

# visual-prompt-eval

> Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want — Lin et al. (2024) (arXiv:2403.20271, 2024)

## What this evaluates

Evaluates multimodal large language models' ability to comprehend and reason about visual prompts (points, bounding boxes, free-form shapes) for fine-grained object classification, region captioning, OCR, and complex visual reasoning.

## Datasets

- **LVIS** — total ?; splits: val (-1)
- **PACO** — total ?; splits: val (-1)
- **COCO-Text** — total ?; splits: val (-1)
- **RefCOCOg** — total ?; splits: val (-1)
- **MDVP-Bench** — total ?; splits: test (-1)
- **LLaVA-Bench** — total ?; splits: test (-1)
- **Ferret-Bench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified objects, recognized text, or answered questions in zero-shot settings.
- `Semantic Similarity` — range: [0, 1]
  - Measures semantic relevance between predicted and ground-truth classifications using word embeddings.
- `Semantic IOU` — range: [0, 1]
  - Computes the intersection over union of semantic embeddings for predicted and ground-truth regions.
- `GPT-4V Score` — range: percent
  - Ratio of the model's GPT-4 evaluation score to the GPT-4 baseline score, expressed as a percentage.
- `METEOR` — range: [0, 1]
  - Metric for evaluating the quality of generated captions against ground truth, considering synonymy and stemming.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores.

## Input / output format

**Input**: Image paired with a visual prompt (point coordinate, bounding box, or free-form shape) and a text instruction/question targeting the prompted region.

**Output**: Text response containing the predicted class label, OCR text, region description, or reasoning answer.

## Scoring recipe

```python
def evaluate(predictions, gold, metric_name):
    if metric_name == 'Accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_name in ['Semantic Similarity', 'Semantic IOU']:
        return compute_embedding_overlap(predictions, gold)
    elif metric_name == 'GPT-4V Score':
        return (model_gpt4_score / baseline_gpt4_score) * 100
    elif metric_name in ['METEOR', 'CIDEr']:
        return standard_caption_metric(predictions, gold)
    return 0.0
```

## Common pitfalls

- Models are evaluated strictly in zero-shot mode without dataset-specific fine-tuning during testing.
- Visual prompts (boxes/points) are sometimes randomly perturbed or scaled to simulate free-form inputs, which can artificially lower classification metrics if not accounted for.
- GPT-4V scoring for detailed captions uses a ratio to a GPT-4 baseline rather than absolute scores, requiring careful normalization.

## Evidence (verbatim from paper)

> In all evaluation experiments, we will not continue to fine-tune on a specific dataset but will instead adopt a zero-shot testing approach. Following (Yuan et al., 2024a), we employ two semantic relevance indicators—Semantic Similarity (SS) and Semantic Intersection over Union (S-IOU)—to assess the model's classification performance... GPT-4 is then used to assess the captions generated by the MLLMs, with evaluation scores ranging from 1 to 10 and calculate the ratio of the predicted score to that of GPT-4, expressed as a percentage.

## Citation

```bibtex
@misc{lin2024drawandunderstand,
  title={Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2403.20271}
}
```

- arXiv: 2403.20271