Blink Vision Centric Eval

---
name: blink-vision-centric-eval
description: This evaluation probes a model's ability to leverage raw visual representations for vision-centric tasks without relying on language priors or domain expertise. It tests pixel-level matching, depth perception, 3D object awareness, and art style recognition across multiple-choice and regression-style tasks. Use when the user wants to benchmark on CV-Bench (Depth Order), SPair-71k, FunKPoint, HPatches, MOCHI, WikiArt (BLINK Art Style), or asks about evaluating this task. Reports multiple-choice VQA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08008
  bibtex_key: fu2025hidden
  confidence: high
---

# blink-vision-centric-eval

> Hidden in plain sight: VLMs overlook their visual representations — Fu et al. (2025) (arXiv:2506.08008, 2025)

## What this evaluates

This evaluation probes a model's ability to leverage raw visual representations for vision-centric tasks without relying on language priors or domain expertise. It tests pixel-level matching, depth perception, 3D object awareness, and art style recognition across multiple-choice and regression-style tasks.

## Datasets

- **CV-Bench (Depth Order)** — total ?; splits: test (-1)
- **SPair-71k** — total ?; splits: test (-1)
- **FunKPoint** — total ?; splits: test (-1)
- **HPatches** — total ?; splits: test (-1)
- **MOCHI** — total ?; splits: test (-1)
- **WikiArt (BLINK Art Style)** — total ?; splits: test (-1)

## Metrics

- `multiple-choice VQA` **(primary)** — range: [0, 1]
  - Percentage of correctly selected options in a multiple-choice VQA format. For vision encoders, task-specific metrics are used: depth comparison accuracy, cosine similarity matching, and MSE between Gram matrices for style.

## Input / output format

**Input**: One or two images (reference/target), optionally with bounding boxes or dots, accompanied by a multiple-choice prompt asking to identify the correct match, closer object, or matching style.

**Output**: A single letter option (e.g., 'A', 'B', 'C', 'D') or a predicted depth/value for encoder evaluation.

## Scoring recipe

```python
def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
# For encoders: compare predicted values (depth, similarity, MSE) against ground truth thresholds or select argmax/argmin.
```

## Common pitfalls

- Assuming strong vision encoder performance implies strong VLM performance; the LLM projection/decoding layer is often the bottleneck.
- Confusing language priors or domain knowledge with actual visual reasoning capabilities.
- Overlooking prompt sensitivity; small prompt changes can drastically alter VLM performance on vision tasks.

## Evidence (verbatim from paper)

> We evaluate VLMs through a multiple-choice VQA format, and evaluate the vision encoder by computing pairwise cosine similarity of CLS embeddings and choosing the example with the lowest average score.

## Citation

```bibtex
@misc{fu2025hidden,
  title={Hidden in plain sight: VLMs overlook their visual representations},
  author={Fu et al. (2025)},
  year={2025},
  note={arXiv:2506.08008}
}
```

- arXiv: 2506.08008