Vl Compositionality Eval

---
name: vl-compositionality-eval
description: Evaluates vision-language models on compositional reasoning capabilities, specifically testing their ability to correctly bind attributes, understand semantic relations, and parse word order in image-text pairs. It also measures systematic generalization to unseen concept combinations and zero-shot classification and retrieval performance. Use when the user wants to benchmark on ARO, CREPE, SVO, VL-Checklist, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.13812
  bibtex_key: singh2023mosaiclip
  confidence: high
---

# vl-compositionality-eval

> Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality — Harman Singh et al. (2023) (arXiv:2305.13812, 2023)

## What this evaluates

Evaluates vision-language models on compositional reasoning capabilities, specifically testing their ability to correctly bind attributes, understand semantic relations, and parse word order in image-text pairs. It also measures systematic generalization to unseen concept combinations and zero-shot classification and retrieval performance.

## Datasets

- **ARO** — total ?; splits: test (-1)
- **CREPE** — total ?; splits: test (-1)
- **SVO** — total ?; splits: test (-1)
- **VL-Checklist** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the percentage of correctly predicted labels (e.g., attribute, relation, word order, or unseen compound/atom) out of the total test instances.

## Input / output format

**Input**: Paired image and text (sentence or phrase) instances.

**Output**: Predicted class label or similarity score for matching/mis-matching pairs, used to compute accuracy or Recall@1.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Fine-tuning on COCO can cause data leakage into CREPE and ARO benchmarks due to image overlap, confounding generalization results.
- Different models use different backbones and pre-training data (OpenAI CLIP vs OpenCLIP), making direct comparison sensitive to the evaluation split and architecture.
- The 'Avg.' column in tables aggregates specific sub-metrics (Relation, Attribute, Order) rather than averaging across all datasets.

## Evidence (verbatim from paper)

> On average, MosaiCLIP achieves +3.3%,+6.3% better performance on the ELEVATER classification benchmark compared to NegCLIP and CLIP while pre-training and maintains similar accuracy as CLIP while fine-tuning.

## Citation

```bibtex
@misc{singh2023mosaiclip,
  title={Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality},
  author={Harman Singh et al. (2023)},
  year={2023},
  note={arXiv:2305.13812}
}
```

- arXiv: 2305.13812