Biocap Eval

---
name: biocap-eval
description: Evaluates zero-shot species classification and fine-grained text-image retrieval capabilities in biological domains. Probes the model's ability to align visual features with taxonomic labels and descriptive natural language without task-specific fine-tuning. Use when the user wants to benchmark on NABirds, Meta-Album (Plankton, Insects, Insects 2), IDLE-OO Camera Traps, Rare Species, PlantNet, Fungi, PlantVillage, Med. Leaf, INQUIRE-Rerank, Cornell Bird, PlantID, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.20095
  bibtex_key: zhang2025biocap
  confidence: high
---

# biocap-eval

> BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models — Ziheng Zhang et al. (arXiv:2510.20095, 2025)

## What this evaluates

Evaluates zero-shot species classification and fine-grained text-image retrieval capabilities in biological domains. Probes the model's ability to align visual features with taxonomic labels and descriptive natural language without task-specific fine-tuning.

## Datasets

- **NABirds** — total ?; splits: test (-1)
- **Meta-Album (Plankton, Insects, Insects 2)** — total ?; splits: test (-1)
- **IDLE-OO Camera Traps** — total ?; splits: test (-1)
- **Rare Species** — total ?; splits: test (-1)
- **PlantNet** — total ?; splits: test (-1)
- **Fungi** — total ?; splits: test (-1)
- **PlantVillage** — total ?; splits: test (-1)
- **Med. Leaf** — total ?; splits: test (-1)
- **INQUIRE-Rerank** — total ?; splits: test (-1)
- **Cornell Bird** — total ?; splits: test (-1)
- **PlantID** — total ?; splits: test (-1)

## Metrics

- `top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted species labels out of the total number of test instances.
- `AP@50` — range: [0, 1]
  - Average Precision computed over the top 50 ranked documents for each query.
- `Recall@10` — range: [0, 1]
  - Fraction of queries where the ground-truth matching image or text appears in the top 10 retrieved results.

## Input / output format

**Input**: RGB image paired with a species name (for classification) or a natural language query (for retrieval).

**Output**: Predicted class label (classification) or a ranked list of images/texts (retrieval).

## Scoring recipe

```python
def compute_metrics(predictions, golds, k=10, max_docs=50):
    # Classification
    acc = sum(p == g for p, g in zip(predictions['cls'], golds['cls'])) / len(golds['cls'])
    
    # Retrieval Recall@K
    recalls = []
    for img, texts, gold_text in zip(predictions['img'], golds['texts'], golds['gold_text']):
        scores = model.compute_similarity(img, texts)
        ranked = argsort(scores, descending=True)[:k]
        recalls.append(any(gold_text == texts[i] for i in ranked))
    recall_at_k = sum(recalls) / len(recalls)
    
    # AP@50
    aps = []
    for query, docs, gold_doc in zip(predictions['query'], golds['docs'], golds['gold_doc']):
        scores = model.compute_similarity(query, docs)
        ranked = argsort(scores, descending=True)[:max_docs]
        relevant = [1 if docs[i] == gold_doc else 0 for i in range(len(ranked))]
        precisions = [sum(relevant[:i+1]) / (i+1) for i in range(len(relevant))]
        ap = sum(p * r for p, r in zip(precisions, relevant)) / max(sum(relevant), 1)
        aps.append(ap)
    ap_at_50 = sum(aps) / len(aps)
    return acc, recall_at_k, ap_at_50
```

## Common pitfalls

- Zero-shot evaluation prohibits fine-tuning on target benchmarks, making results highly sensitive to pre-training data overlap and domain shift.
- Retrieval performance is asymmetric; I2T and T2I Recall@10 are reported separately and can diverge significantly due to modality mismatch.
- Averaging accuracy across 10 heterogeneous benchmarks (birds, plants, fungi, camera traps) may obscure domain-specific failures or dataset scale imbalances.

## Evidence (verbatim from paper)

> We evaluate the natural language understanding on INQUIRE-Rerank (AP@50), Cornell Bird and PlantID(Recall@10) in [Table 2].

## Citation

```bibtex
@misc{zhang2025biocap,
  title={BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models},
  author={Ziheng Zhang et al.},
  year={2025},
  note={arXiv:2510.20095}
}
```

- arXiv: 2510.20095