Fine Grained Classification Eval

---
name: fine-grained-classification-eval
description: Probes the ability of vision-language models to distinguish visually similar subcategories within broader classes (e.g., specific bird species or car models) using a multiple-choice format. Use when the user wants to benchmark on ImageNet-1K, Oxford Flowers-102, Oxford-IIIT Pet-37, Food-101, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.17871
  bibtex_key: ghosh2026finegrained
  confidence: high
---

# fine-grained-classification-eval

> Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models — Ghosh et al. (2026) (arXiv:2602.17871, 2026)

## What this evaluates

Probes the ability of vision-language models to distinguish visually similar subcategories within broader classes (e.g., specific bird species or car models) using a multiple-choice format.

## Datasets

- **ImageNet-1K** — total ?; splits: test (-1)
- **Oxford Flowers-102** — total ?; splits: test (-1)
- **Oxford-IIIT Pet-37** — total ?; splits: test (-1)
- **Food-101** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match accuracy: the proportion of test images where the model's predicted option exactly matches the ground-truth class label.

## Input / output format

**Input**: An image paired with five multiple-choice options (A–E), where one option is the ground-truth class and the other four are hard negatives selected via OpenCLIP ViT-L/14 cosine similarity.

**Output**: A single letter (A, B, C, D, or E) corresponding to the selected class option.

## Scoring recipe

```python
correct = 0
total = 0
for img, options, ground_truth in dataset:
    pred = model.generate(img, options)
    if pred == ground_truth:
        correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- The multiple-choice options are dynamically generated per image using OpenCLIP cosine similarity rather than being fixed across the dataset.
- Performance is measured on exact match to the option letter, not semantic similarity or free-text generation.

## Evidence (verbatim from paper)

> We evaluate the fine-grained visual knowledge of VLMs using four well-established object recognition benchmarks, each focusing on different domains of fine-grained classification: ImageNet-1K, Oxford Flowers-102, Oxford-IIIT Pet-37, and Food-101. For each VLM, we use the respective default prompt format from VLMEvalKit and measure accuracy based on the exact match with the multiple choice options.

## Citation

```bibtex
@misc{ghosh2026finegrained,
  title={Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models},
  author={Ghosh et al. (2026)},
  year={2026},
  note={arXiv:2602.17871}
}
```

- arXiv: 2602.17871