Commonsense Retrieval Eval

---
name: commonsense-retrieval-eval
description: This benchmark probes the commonsense reasoning capabilities of vision-language models by evaluating their ability to match images to text riddles (or vice versa) where the subject entity is replaced with a demonstrative pronoun. It specifically tests relational knowledge retrieval and generalization to unseen knowledge triples. Use when the user wants to benchmark on DANCE Diagnostic Set, or asks about evaluating this task. Reports Acc@50.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.16504
  bibtex_key: ye2022improving
  confidence: high
---

# commonsense-retrieval-eval

> Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles — Ye et al. (2022) (arXiv:2211.16504, 2022)

## What this evaluates

This benchmark probes the commonsense reasoning capabilities of vision-language models by evaluating their ability to match images to text riddles (or vice versa) where the subject entity is replaced with a demonstrative pronoun. It specifically tests relational knowledge retrieval and generalization to unseen knowledge triples.

## Datasets

- **DANCE Diagnostic Set** — total ?; splits: train (-1), test-seen (-1), test-unseen (-1); repo https://github.com/pleaseconnectwifi/DANCE

## Metrics

- `Acc@50` **(primary)** — range: [0, 1]
  - Models compute alignment scores between a query (riddle or image) and a candidate set of 50 items. The candidates are sorted by score, and accuracy is calculated as the fraction of queries where the ground-truth match appears in the top 50 predictions.

## Input / output format

**Input**: A text riddle with a hidden subject (or an image) paired with a candidate list of 50 images (or riddles).

**Output**: A ranked list of the 50 candidate images (or riddles) sorted by the model's alignment score.

## Scoring recipe

```python
def compute_acc50(predictions, gold):
    correct = 0
    for pred_list, gt_idx in zip(predictions, gold):
        if gt_idx in pred_list[:50]:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Hard negatives are constructed using ConceptNet graph distance (RelatedTo, DistinctFrom, Antonym) rather than random sampling, making the evaluation sensitive to semantic entity correlation rather than just visual similarity.
- The test set is divided into 'test-seen' and 'test-unseen' splits based on knowledge triple presence in training, but all images are strictly held out from the training set to prevent memorization.
- Perplexity is used as the underlying scoring mechanism for alignment, but the reported metric is strictly top-50 accuracy, not perplexity values.

## Evidence (verbatim from paper)

> For automatic evaluation, we adopt perplexity score as the evaluation metric, following the works*[[13](#bib.bib13 ""), [50](#bib.bib50 "")]*. In the experiment, we set the candidate number for each sample as $50$, with the number of positive samples $n$ between $1$ to $15$, and measure the average accuracy of retrieved ground truth inference, denoted as $Acc@50$.

## Citation

```bibtex
@misc{ye2022improving,
  title={Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2211.16504}
}
```

- arXiv: 2211.16504