Biological Visual Eval

---
name: biological-visual-eval
description: Evaluates frozen visual embedding extractors on ecological trait alignment, fine-grained intra-species variation preservation, and zero-shot/few-shot transfer learning across diverse biological domains. Use when the user wants to benchmark on FishNet, NeWT, AwA2, Herb, PlantDoc, Life stage-Diff/Align, Sex-Diff/Align, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23883
  bibtex_key: gu2025bioclip2
  confidence: high
---

# biological-visual-eval

> BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning — Jianyang Gu et al. (2025) (arXiv:2505.23883, 2025)

## What this evaluates

Evaluates frozen visual embedding extractors on ecological trait alignment, fine-grained intra-species variation preservation, and zero-shot/few-shot transfer learning across diverse biological domains.

## Datasets

- **FishNet** — total 94532; splits: train (75631), test (18901)
- **NeWT** — total ?; splits: train (-1), test (-1)
- **AwA2** — total 37322; splits: train (-1), test (-1)
- **Herb** — total ?; splits: train (-1), test (-1)
- **PlantDoc** — total 2598; splits: train (-1), test (-1)
- **Life stage-Diff/Align** — total ?; splits: train (-1), test (-1)
- **Sex-Diff/Align** — total 13624; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted instances. For FishNet, a sample is counted correct only if all 9 binary labels are predicted correctly.
- `average F1 score` — range: [0, 1]
  - Macro-averaged F1 score computed across all 85 numeric attributes in AwA2.
- `clustering accuracy` — range: [0, 1]
  - Accuracy of semi-supervised K-means cluster assignments compared to ground-truth species labels.

## Input / output format

**Input**: Frozen visual embeddings extracted from input images using the evaluated model.

**Output**: Predicted class labels (binary or multi-class), attribute scores, or cluster assignments.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name == 'accuracy':
        if predictions.ndim > 1:  # FishNet multi-label
            correct = (predictions == gold).all(axis=1)
        else:
            correct = (predictions == gold)
        return correct.mean()
    elif metric_name == 'average F1 score':
        return f1_score(gold, predictions, average='macro')
    elif metric_name == 'clustering accuracy':
        return hungarian_algorithm_accuracy(gold, predictions)
    return 0.0
```

## Common pitfalls

- Fine-tuning the vision model instead of using it as a frozen embedding extractor, which violates the protocol's design to isolate embedding quality.
- Reporting per-label accuracy for FishNet instead of requiring all 9 binary labels to be correct simultaneously for a sample.
- Using the full AwA2 class split instead of the specified 45 train / 5 unseen test zero-shot transfer setting.

## Evidence (verbatim from paper)

> The average F1 score over all the attributes is reported for this benchmark.

## Citation

```bibtex
@misc{gu2025bioclip2,
  title={BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning},
  author={Jianyang Gu et al. (2025)},
  year={2025},
  note={arXiv:2505.23883}
}
```

- arXiv: 2505.23883