Do You See Me Eval

---
name: do-you-see-me-eval
description: This benchmark isolates and evaluates core visual perception capabilities in multimodal large language models (MLLMs) through programmatically generated tasks inspired by human psychology. It probes figure-ground discrimination, spatial relations, visual form constancy, and perceptual brittleness across 2D and 3D settings with controlled difficulty levels. Use when the user wants to benchmark on Do You See Me, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02022
  bibtex_key: kanade2025doyouseeme
  confidence: high
---

# do-you-see-me-eval

> Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs — Kanade et al. (2025) (arXiv:2506.02022, 2025)

## What this evaluates

This benchmark isolates and evaluates core visual perception capabilities in multimodal large language models (MLLMs) through programmatically generated tasks inspired by human psychology. It probes figure-ground discrimination, spatial relations, visual form constancy, and perceptual brittleness across 2D and 3D settings with controlled difficulty levels.

## Datasets

- **Do You See Me** — total ?; splits: test (-1); repo https://github.com/microsoft/Do-You-See-Me

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions after parsing the model's extended output into a standardized label or value. Calculated as the number of correct predictions divided by the total number of samples.

## Input / output format

**Input**: An image paired with a predefined textual prompt containing a perception question (and optionally a reasoning question for the joint dataset), instructing the model to generate a detailed chain-of-thought response.

**Output**: A detailed textual response containing chain-of-thought reasoning, which is subsequently parsed by an answer extractor (GPT-4o) into a concise standardized answer (e.g., multiple-choice label or numeric value).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    # predictions: list of raw model outputs
    # gold_labels: list of ground truth answers
    extracted = [extract_answer(pred) for pred in predictions]  # GPT-4o parser
    standardized = [standardize(extracted_i) for extracted_i in extracted]  # to MCQ label or number
    correct = sum(1 for pred, gold in zip(standardized, gold_labels) if pred == gold)
    return correct / len(gold_labels)
```

## Common pitfalls

- Models can arrive at correct final answers despite misperceiving crucial visual elements (observed in 29% of correct responses).
- Chain-of-Thought prompting degrades performance by introducing lossy text representations of visual content.
- Models may exploit multiple-choice cues or prompt structure rather than performing genuine visual analysis.

## Evidence (verbatim from paper)

> Given that modern LLMs and MLLMs often generate extended textual outputs, making traditional rule-based or template-matching evaluation methods challenging*[[24]]*, we employ an expert LLM-based evaluation protocol, inspired by recent benchmarks*[[25], [47]]*. This framework proceeds in three stages: First, the MLLM under evaluation generates a detailed response following a predefined input template (see [Appendix J] for prompt structures). Second, an answer extractor, specifically GPT-4o *[[26]]* (the same version as evaluated), parses these extended outputs to yield a concise answer. This approach leverages the high accuracy of expert LLMs in answer extraction, as demonstrated in prior work*[[25]]*. Third, the extracted text is standardized (e.g., reduced to multiple-choice labels or numeric values). Performance across all subtasks is measured using accuracy.

## Citation

```bibtex
@misc{kanade2025doyouseeme,
  title={Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs},
  author={Kanade et al. (2025)},
  year={2025},
  note={arXiv:2506.02022}
}
```

- arXiv: 2506.02022