Bioscan 5m Eval

---
name: bioscan-5m-eval
description: Evaluates models on insect biodiversity monitoring by testing closed-world species identification, open-world genus-level grouping for novel species, and zero-shot clustering of multimodal embeddings against taxonomic ground truth. Use when the user wants to benchmark on BIOSCAN-5M, or asks about evaluating this task. Reports Fine-tuned accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.12723
  bibtex_key: gharaee2024bioscan5m
  confidence: high
---

# bioscan-5m-eval

> BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity — Gharaee et al. (2024) (arXiv:2406.12723, 2024)

## What this evaluates

Evaluates models on insect biodiversity monitoring by testing closed-world species identification, open-world genus-level grouping for novel species, and zero-shot clustering of multimodal embeddings against taxonomic ground truth.

## Datasets

- **BIOSCAN-5M** — total 5000000; splits: pretrain (4677756), train (289203), val (14757), test (39373), key_unseen (36465), val_unseen (8819), test_unseen (7887), other_heldout (76590); repo https://github.com/bioscan-m1/BIOSCAN-5M

## Metrics

- `Fine-tuned accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted species labels on the test split after full model fine-tuning.
- `Linear probe accuracy` — range: percent
  - Accuracy of a linear classifier trained on frozen pretrained embeddings to predict species labels on the test split.
- `1NN-Probe accuracy` — range: percent
  - Genus-level accuracy on unseen species test set using nearest-neighbor classification on averaged token embeddings, fitted on seen species.
- `Adjusted Mutual Information (AMI)` — range: [0, 1]
  - Clustering agreement score between predicted clusters and ground-truth taxonomic labels, normalized by entropy of true labels.
- `Top-1 macro accuracy` — range: percent
  - Macro-averaged top-1 accuracy for taxonomic classification across modalities in the multimodal retrieval setting.

## Input / output format

**Input**: DNA barcode sequences (strings), high-resolution insect images, and taxonomic text labels.

**Output**: Predicted taxonomic labels (species/genus), cluster assignments, or retrieved key indices.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold) * 100

def compute_1nn_probe(seen_embeds, seen_labels, unseen_seqs, model):
    unseen_preds = []
    for seq in unseen_seqs:
        emb = avg_tokens(model.encode(seq))
        dists = [np.linalg.norm(emb - s) for s in seen_embeds]
        unseen_preds.append(seen_labels[np.argmin(dists)])
    return compute_accuracy(unseen_preds, unseen_labels)

def compute_ami(true_labels, predictions):
    from sklearn.metrics import adjusted_mutual_info_score
    return adjusted_mutual_info_score(true_labels, predictions)
```

## Common pitfalls

- Splits are partitioned by barcode to prevent data leakage across train/val/test sets, meaning all samples sharing a barcode stay together.
- Test set species distribution is flattened to avoid imbalance, unlike the natural dataset distribution.
- Embedding dimensions vary across models (e.g., 128 vs 512 vs 768), which can unfairly impact 1NN probing performance.
- Zero-shot clustering uses UMAP dimensionality reduction to 50D before Agglomerative Clustering, which may obscure fine-grained taxonomic structure.

## Evidence (verbatim from paper)

> Evaluate against the ground-truth annotations with Adjusted Mutual Information (AMI) score (Vinh et al., 2010), measuring the percentage information explained relative to the entropy of the true labels.

## Citation

```bibtex
@misc{gharaee2024bioscan5m,
  title={BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity},
  author={Gharaee et al. (2024)},
  year={2024},
  note={arXiv:2406.12723}
}
```

- arXiv: 2406.12723