Vit Zero Shot Clustering Eval

---
name: vit-zero-shot-clustering-eval
description: Evaluates the ability of Vision Transformer models combined with dimensionality reduction and clustering algorithms to perform zero-shot species-level clustering of animal images. It probes how well unsupervised pipelines can recover ground-truth taxonomic labels and capture intra-specific variation without manual annotation. Use when the user wants to benchmark on Animal Images (Birds & Mammals), or asks about evaluating this task. Reports V-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.03894
  bibtex_key: markoff2026vitclustering
  confidence: high
---

# vit-zero-shot-clustering-eval

> Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study — Hugo Markoff, Stefan Hein Bengtson, Michael Ørsted (2026) (arXiv:2602.03894, 2026)

## What this evaluates

Evaluates the ability of Vision Transformer models combined with dimensionality reduction and clustering algorithms to perform zero-shot species-level clustering of animal images. It probes how well unsupervised pipelines can recover ground-truth taxonomic labels and capture intra-specific variation without manual annotation.

## Datasets

- **Animal Images (Birds & Mammals)** — total ?; splits: test (-1)

## Metrics

- `V-measure` **(primary)** — range: [0, 1]
  - Harmonic mean of clustering homogeneity and completeness with respect to ground-truth species labels. Values closer to 1 indicate perfect alignment between predicted clusters and true species.
- `Isolation Index (II)` — range: [0, 1]
  - Measures average purity per species: II_s = (1/N_s) * sum_{c in C} (n_{s,c}^2 / |c|), where N_s is total images of species s, n_{s,c} is count of species s in cluster c, and |c| is cluster size.
- `Effective Cluster Count (ECC)` — range: other
  - Measures how many clusters a species effectively 'owns': ECC_s = sum_{c in C} (n_{s,c} / |c|). Values near 1 indicate a species forms a single pure cluster.

## Input / output format

**Input**: Unlabeled animal images (RGB, IR, or flash) representing multiple species.

**Output**: Discrete cluster labels assigned to each image.

## Scoring recipe

```python
def compute_vmeasure(predictions, ground_truth):
    return v_measure_score(ground_truth, predictions)

def compute_ii_ecc(predictions, ground_truth):
    clusters = defaultdict(list)
    for img_id, cluster_id in enumerate(predictions):
        clusters[cluster_id].append(ground_truth[img_id])
    ii_scores, ecc_scores = [], []
    for species in set(ground_truth):
        n_s = sum(1 for g in ground_truth if g == species)
        ii_s = sum((sum(1 for g in c if g == species)**2) / len(c) for c in clusters.values()) / n_s
        ecc_s = sum(sum(1 for g in c if g == species) / len(c) for c in clusters.values())
        ii_scores.append(ii_s)
        ecc_scores.append(ecc_s)
    return mean(ii_scores), mean(ecc_scores)
```

## Common pitfalls

- Assuming the number of predicted clusters equals the true number of species; unsupervised methods like DBSCAN heavily over-split, while HDBSCAN slightly over-estimates.
- Interpreting cluster oversplitting as failure; intra-specific variation (age, sex, lighting) naturally fragments species into multiple pure clusters, which does not compromise identification accuracy.
- Applying supervised clustering parameters (e.g., fixed K=30) in zero-shot scenarios where the true species count is unknown.

## Evidence (verbatim from paper)

> DINOv3 achieves the highest average V-measure (0.817), followed by DINOv2 (0.769). In contrast, biology-specific BioCLIP 2 and general-purpose CLIP and SigLIP show substantially lower performance, with average V-measures ranging from 0.597 to 0.652.

## Citation

```bibtex
@misc{markoff2026vitclustering,
  title={Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study},
  author={Hugo Markoff, Stefan Hein Bengtson, Michael Ørsted (2026)},
  year={2026},
  note={arXiv:2602.03894}
}
```

- arXiv: 2602.03894