Fish Vista Eval

---
name: fish-vista-eval
description: Evaluates computer vision models on fine-grained species classification, multi-label trait identification, and pixel-level trait segmentation in fish images. Probes capabilities in handling long-tailed distributions, out-of-distribution generalization to unseen species, and localizing small/rare anatomical features. Use when the user wants to benchmark on Fish-Vista, or asks about evaluating this task. Reports macro-averaged F1-score, Mean Average Precision (mAP), mean Intersection over Union (mIoU).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.08027
  bibtex_key: mehrab2024fishvista
  confidence: high
---

# fish-vista-eval

> Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images — Mehrab et al. (2024) (arXiv:2407.08027, 2024)

## What this evaluates

Evaluates computer vision models on fine-grained species classification, multi-label trait identification, and pixel-level trait segmentation in fish images. Probes capabilities in handling long-tailed distributions, out-of-distribution generalization to unseen species, and localizing small/rare anatomical features.

## Datasets

- **Fish-Vista** — total 69126; splits: train (-1), test (-1), in-species (-1), leave-out-species (-1), manual-annotated (-1)

## Metrics

- `macro-averaged F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall calculated per class and then averaged across all species classes, robust to class imbalance.
- `Mean Average Precision (mAP)` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve averaged across all trait classes, evaluating multi-label classification performance.
- `mean Intersection over Union (mIoU)` **(primary)** — range: [0, 1]
  - Average of IoU scores computed per trait class, where IoU is the intersection area divided by the union area between predicted and ground-truth segmentation masks.

## Input / output format

**Input**: RGB images of fish specimens against uniform backgrounds.

**Output**: Per instance: (1) Predicted species class label, (2) Binary vector indicating presence/absence of four anatomical traits, (3) Pixel-level segmentation mask for each trait.

## Scoring recipe

```python
def evaluate(dataset, model):
    # Classification & Identification
    f1 = macro_f1_score(dataset.y_species, model.predict_species(dataset.X))
    mAP = mean_average_precision(dataset.y_traits, model.predict_traits(dataset.X))
    
    # Segmentation
    ious = []
    for trait in dataset.traits:
        pred_mask = model.predict_segmentation(dataset.X, trait)
        gt_mask = dataset.masks[trait]
        intersection = np.logical_and(pred_mask, gt_mask).sum()
        union = np.logical_or(pred_mask, gt_mask).sum()
        ious.append(intersection / union if union > 0 else 0.0)
    mIoU = np.mean(ious)
    return {'F1': f1, 'mAP': mAP, 'mIoU': mIoU}
```

## Common pitfalls

- Long-tailed class distribution causes standard models to drop to ~50% accuracy on ultra-rare species despite high performance on majority classes.
- High image-level identification accuracy does not guarantee correct spatial localization, as attention maps often fail to align with ground-truth trait masks.
- Small, rare traits (e.g., adipose fin, barbel) are frequently misclassified as larger neighboring traits due to low pixel area and visual similarity.

## Evidence (verbatim from paper)

> We report the overall macro-averaged F1-score and mean class accuracy for each subcategory to assess performance across the imbalanced distribution. ... Figure [6] compares the Mean Average Precision (mAP) of top-performing models across all test sets. ... Table [4] presents the overall mIoU and individual trait-wise IoUs for each method.

## Citation

```bibtex
@misc{mehrab2024fishvista,
  title={Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images},
  author={Mehrab et al. (2024)},
  year={2024},
  note={arXiv:2407.08027}
}
```

- arXiv: 2407.08027