Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs. Use when the user wants to benchmark on ImageNetVC, or asks about evaluating this task. Reports Top-1 accuracy.
Installs into .claude/skills of the current project.
Are you the author of Imagenetvc Eval?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-imagenetvc-eval)
---
name: imagenetvc-eval
description: Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs. Use when the user wants to benchmark on ImageNetVC, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
skill_kind: dataset_eval
source_arxiv: 2305.15028
bibtex_key: [REDACTED]
confidence: high
---
# imagenetvc-eval
> ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories — Heming Xia et al. (2023) (arXiv:2305.15028, 2023)
## What this evaluates
Evaluates zero- and few-shot visual commonsense reasoning capabilities of language models and visually-augmented language models across 1,000 ImageNet categories using human-annotated QA pairs.
## Datasets
- **ImageNetVC** — total 1000; splits: test (-1); repo https://github.com/hemingkx/ImageNetVC
## Metrics
- `Top-1 accuracy` **(primary)** — range: percent
- Fraction of correct predictions. Model scores each candidate answer y as 1 / -log P(y|context), applies softmax over all candidates to get probabilities, and selects the argmax. Accuracy is the percentage of instances where the predicted answer matches the ground truth.
## Input / output format
**Input**: For LLMs: text prompt formatted as '[Question] The answer is [Answer].' For VaLMs: same text prompt concatenated with top-K (K=10) images retrieved via Google Image Search or synthesized via Stable Diffusion, sorted by CLIP similarity to the question.
**Output**: Probability distribution over all candidate answers via softmax. Final prediction is the argmax of this distribution.
## Scoring recipe
```python
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return (correct / len(gold)) * 100
# Per-instance prediction:
# log_probs = {y: -math.log(model.log_prob(y, context)) for y in candidates}
# probs = softmax(log_probs)
# pred = argmax(probs)
```
## Common pitfalls
- Prompt format bias can skew raw log-probabilities; the paper uses calibration/normalization to mitigate this.
- Performance variance is high across different prompt templates; results should be averaged over multiple prompts.
- VaLM evaluation depends heavily on image retrieval/synthesis quality and CLIP-based sorting, which may not perfectly align with the question.
## Evidence (verbatim from paper)
> Following Schick and Schütze ([2021](#bib.bib28 "")) and Yang et al. ([2022](#bib.bib35 "")), we treat the zero-shot evaluation as a cloze test, transforming the QA pairs in ImageNetVC into prompts like “[Question] The answer is [Answer].” ... The numbers along the radio axis denote the mean Top-1 accuracy (%) of models over 5 different prompts.
## Citation
```bibtex
@misc{xia2023imagenetvc,
title={ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories},
author={Heming Xia et al. (2023)},
year={2023},
note={arXiv:2305.15028}
}
```
- arXiv: 2305.15028
