Imagenetvc Eval

---
name: imagenetvc-eval
description: Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs. Use when the user wants to benchmark on ImageNetVC, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.15028
  bibtex_key: xia2023imagenetvc
  confidence: high
---

# imagenetvc-eval

> ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories — Heming Xia et al. (2023) (arXiv:2305.15028, 2023)

## What this evaluates

Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs.

## Datasets

- **ImageNetVC** — total 1000; splits: test (-1); repo https://github.com/hemingkx/ImageNetVC

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Fraction of correct predictions. Model scores each candidate answer y as 1 / -log P(y|context), applies softmax over all candidates to get probabilities, and selects the argmax. Accuracy is the percentage of instances where the predicted answer matches the ground truth.

## Input / output format

**Input**: For LLMs: text prompt formatted as '[Question] The answer is [Answer].' For VaLMs: same text prompt concatenated with top-K (K=10) images retrieved via Google Image Search or synthesized via Stable Diffusion, sorted by CLIP similarity to the question.

**Output**: Probability distribution over all candidate answers via softmax. Final prediction is the argmax of this distribution.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

# Per-instance prediction:
# log_probs = {y: -math.log(model.log_prob(y, context)) for y in candidates}
# probs = softmax(log_probs)
# pred = argmax(probs)
```

## Common pitfalls

- Prompt format bias can skew raw log-probabilities; the paper uses calibration/normalization to mitigate this.
- Performance variance is high across different prompt templates; results should be averaged over multiple prompts.
- VaLM evaluation depends heavily on image retrieval/synthesis quality and CLIP-based sorting, which may not perfectly align with the question.

## Evidence (verbatim from paper)

> Following Schick and Schütze ([2021](#bib.bib28 "")) and Yang et al. ([2022](#bib.bib35 "")), we treat the zero-shot evaluation as a cloze test, transforming the QA pairs in ImageNetVC into prompts like “[Question] The answer is [Answer].” ... The numbers along the radio axis denote the mean Top-1 accuracy (%) of models over 5 different prompts.

## Citation

```bibtex
@misc{xia2023imagenetvc,
  title={ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories},
  author={Heming Xia et al. (2023)},
  year={2023},
  note={arXiv:2305.15028}
}
```

- arXiv: 2305.15028