Svo Probes Eval

---
name: svo-probes-eval
description: Evaluates how well CLIP aligns text and images by measuring its preference for positive over negative image-text pairs, and analyzes how semantic features (part-of-speech, concreteness, length, frequency, ambiguity) influence this alignment. Use when the user wants to benchmark on SVO-Probes, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.18786
  bibtex_key: castro2023scalable
  confidence: high
---

# svo-probes-eval

> Scalable Performance Analysis for Vision-Language Models — Castro et al. (2023) (arXiv:2305.18786, 2023)

## What this evaluates

Evaluates how well CLIP aligns text and images by measuring its preference for positive over negative image-text pairs, and analyzes how semantic features (part-of-speech, concreteness, length, frequency, ambiguity) influence this alignment.

## Datasets

- **SVO-Probes** — total ?; splits: test (-1); repo https://github.com/MichiganNLP/Scalable-VLM-Probing

## Metrics

- `accuracy` **(primary)** — range: percent
  - Computed as the percentage of instances where CLIP assigns a higher similarity score to the positive image-text pair than to the negative image-text pair. Evaluated separately for subjects, objects, and verbs.
- `score difference (D)` — range: other
  - D = P - N, where P is the CLIP similarity score for the positive pair and N is the score for the negative pair. Used to measure relative preference strength and feature importance via mean difference.

## Input / output format

**Input**: Pairs of images and text captions (one positive match, one negative mismatch) fed into CLIP to compute similarity scores.

**Output**: Continuous similarity scores from CLIP for each image-text pair.

## Scoring recipe

```python
P = clip_similarity(image_positive, text_positive)
N = clip_similarity(image_negative, text_negative)
D = P - N
is_correct = 1 if P > N else 0

# Accuracy per POS category:
accuracy = sum(is_correct for category in POS) / total_instances_in_category

# Feature importance:
feature_importance = mean(D when feature_present) - mean(D when feature_absent)
```

## Common pitfalls

- Treating the negative image as purely random rather than adversarial, as it shares common elements with the positive image, which inflates N scores.
- Interpreting high absolute CLIP scores as good semantic understanding, when the model often behaves like a bag-of-words model and scores both positive and negative pairs highly if they share vocabulary.
- Confusing the relative preference metric D with absolute retrieval accuracy; D measures preference strength, not correctness probability.

## Evidence (verbatim from paper)

> When computing the number of times CLIP assigns a higher score to the similarity between the text and the positive image as compared to the similarity between the text and the negative image, the verbs obtain 81.45% accuracy while the subjects get 86.87% and the objects 88.78%.

## Citation

```bibtex
@misc{castro2023scalable,
  title={Scalable Performance Analysis for Vision-Language Models},
  author={Castro et al. (2023)},
  year={2023},
  note={arXiv:2305.18786}
}
```

- arXiv: 2305.18786