Mushroom Segmentation Eval

---
name: mushroom-segmentation-eval
description: This benchmark evaluates the zero-shot instance segmentation capability of models trained on synthetic data when applied to real-world agricultural scenes. It probes the model's ability to generalize across domain gaps, handling varying lighting, mushroom densities, and developmental stages without fine-tuning on real annotations. Use when the user wants to benchmark on Real-world on-field dataset, M18K, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08747
  bibtex_key: karoly2025mushroomsegmentation
  confidence: high
---

# mushroom-segmentation-eval

> A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation — Károly et al. (2025) (arXiv:2512.08747, 2025)

## What this evaluates

This benchmark evaluates the zero-shot instance segmentation capability of models trained on synthetic data when applied to real-world agricultural scenes. It probes the model's ability to generalize across domain gaps, handling varying lighting, mushroom densities, and developmental stages without fine-tuning on real annotations.

## Datasets

- **Real-world on-field dataset** — total 195; splits: test (195)
- **M18K** — total 20; splits: test (20)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Used as the primary metric for model selection and comparison.
- `Average Precision (AP)` — range: [0, 1]
  - Area under the precision-recall curve across multiple IoU thresholds or confidence thresholds, measuring detection accuracy.
- `Average Recall (AR)` — range: [0, 1]
  - Area under the recall curve across multiple IoU thresholds or confidence thresholds, measuring the model's ability to find all ground-truth instances.
- `mean Intersection over Union (mIoU)` — range: [0, 1]
  - Average IoU across all predicted and ground-truth instance masks: IoU = |Prediction ∩ Ground Truth| / |Prediction ∪ Ground Truth|.
- `Fréchet Inception Distance (FID)` — range: other
  - Statistical distance between feature distributions of real and generated images using a pre-trained Inception network. Lower values indicate higher visual similarity.
- `Kernel Inception Distance (KID)` — range: other
  - Squared maximum mean discrepancy between feature distributions of real and generated images using a polynomial kernel. Lower values indicate higher visual similarity.

## Input / output format

**Input**: RGB images of white button mushrooms at 512x512 or 1024x1024 resolution, captured in real-world agricultural settings or from the M18K dataset.

**Output**: Instance-level segmentation masks and bounding boxes for each detected mushroom, along with confidence scores.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    ious = compute_iou(predictions, ground_truth)
    tp = count_true_positives(ious)
    fp = len(predictions) - tp
    fn = len(ground_truth) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    ap = average_precision_curve(predictions, ground_truth)
    ar = average_recall_curve(predictions, ground_truth)
    miou = mean(ious)
    return {'F1': f1, 'AP': ap, 'AR': ar, 'mIoU': miou}
```

## Common pitfalls

- Incomplete labeling in the M18K test set causes correctly detected mushrooms to be counted as false positives, artificially lowering AP.
- FID and KID scores measure visual similarity but do not reliably correlate with downstream segmentation performance or human preference.
- Models trained on synthetic data may exhibit conservative detection behavior, favoring precision over recall on datasets with many small or ambiguous objects.

## Evidence (verbatim from paper)

> For model comparison, we report standard instance segmentation metrics: Average Precision (AP), Average Recall (AR), F1 score, and mean Intersection over Union (mIoU). The results of the first experiment, comparing the MRCNN-B and MRCNN-SD models on our real-world dataset, are presented in Table [1].

## Citation

```bibtex
@misc{karoly2025mushroomsegmentation,
  title={A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation},
  author={Károly et al. (2025)},
  year={2025},
  note={arXiv:2512.08747}
}
```

- arXiv: 2512.08747