Omniview Zero Shot Eval

---
name: omniview-zero-shot-eval
description: This evaluation probes the viewpoint invariance and robustness of vision-language pre-training models. It measures how well models maintain classification accuracy on clean data, common out-of-distribution shifts, and specifically challenging viewpoint-variant images compared to standard baselines. Use when the user wants to benchmark on ImageNet-1K, ImageNet-V+, ImageNet-V, OOD-CV, MIRO, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.12139
  bibtex_key: ruan2024omniviewtuning
  confidence: high
---

# omniview-zero-shot-eval

> Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models — Shouwei Ruan et al. (2024) (arXiv:2404.12139, 2024)

## What this evaluates

This evaluation probes the viewpoint invariance and robustness of vision-language pre-training models. It measures how well models maintain classification accuracy on clean data, common out-of-distribution shifts, and specifically challenging viewpoint-variant images compared to standard baselines.

## Datasets

- **ImageNet-1K** — total ?; splits: test (-1)
- **ImageNet-V+** — total ?; splits: test (-1)
- **ImageNet-V** — total ?; splits: test (-1)
- **OOD-CV** — total ?; splits: test (-1)
- **MIRO** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Calculated as the number of correctly predicted samples divided by the total number of samples in the benchmark, multiplied by 100.

## Input / output format

**Input**: Single RGB image per instance.

**Output**: Predicted class label (and confidence scores).

## Scoring recipe

```python
correct = 0
for img, gold_label in dataset:
    logits = model.encode(img)
    pred_label = argmax(logits)
    if pred_label == gold_label:
        correct += 1
top1_acc = (correct / len(dataset)) * 100
```

## Common pitfalls

- Confusing viewpoint-OOD benchmarks (e.g., ImageNet-V, MIRO) with standard distribution-shift benchmarks (e.g., ImageNet-V2, ImageNet-Sketch), which measure different robustness properties.
- Failing to follow the standard CLIP prompting and candidate category name conventions, which significantly impacts zero-shot classification performance.
- Reporting average accuracy across all benchmarks without separating clean, common-OOD, and viewpoint-OOD domains, obscuring the model's specific viewpoint invariance gains.

## Evidence (verbatim from paper)

> Datasets and Metrics. We employ a various set of benchmarks for evaluation, including clean data distributions (ImageNet [13] and CIFAR [28]), common 2D-OOD (ImageNet-V2 [43], ImageNet-Sketch [56], ImageNet-O [22], ImageNet-R [20] and OOD-CV [60]), and most importantly, viewpoint-OOD (ImageNet-V [15], ImageNet-V+ [47], OOD-CV(Pose) [60] and MIRO [7]) datasets. For each benchmark, we report Top-1 and Top-5 accuracy and average accuracy across all benchmarks. The evaluations follow the standard prompting engineering and candidate category names conventions of CLIP [41].

## Citation

```bibtex
@misc{ruan2024omniviewtuning,
  title={Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models},
  author={Shouwei Ruan et al. (2024)},
  year={2024},
  note={arXiv:2404.12139}
}
```

- arXiv: 2404.12139