Croma Eval

---
name: croma-eval
description: Evaluates self-supervised remote sensing representations across classification and segmentation tasks using optical and radar-optical inputs. Probes representation quality via finetuning, linear/nonlinear probing, kNN, and clustering. Use when the user wants to benchmark on BigEarthNet, fMoW-Sentinel, EuroSAT, Canadian Cropland, DFC2020, DW-Expert, MARIDA, or asks about evaluating this task. Reports mAP, Top 1 Acc., mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.00566
  bibtex_key: fuller2023croma
  confidence: high
---

# croma-eval

> CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders — Fuller et al. (2023) (arXiv:2311.00566, 2023)

## What this evaluates

Evaluates self-supervised remote sensing representations across classification and segmentation tasks using optical and radar-optical inputs. Probes representation quality via finetuning, linear/nonlinear probing, kNN, and clustering.

## Datasets

- **BigEarthNet** — total 153485; splits: train (35420), val (118065)
- **fMoW-Sentinel** — total 156226; splits: train (71287), val (84939)
- **EuroSAT** — total 21600; splits: train (16200), val (5400)
- **Canadian Cropland** — total 76972; splits: train (53884), val (23088)
- **DFC2020** — total 55026; splits: train (46152), val (8874)
- **DW-Expert** — total 71444; splits: train (20422), val (51022)
- **MARIDA** — total 3297; splits: train (1682), val (1615)

## Metrics

- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision across all classes. Computes the average precision for each class and averages them, suitable for multi-label classification.
- `Top 1 Acc.` **(primary)** — range: [0, 1]
  - Top-1 Accuracy. The fraction of samples where the predicted class matches the ground truth label.
- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union. Computes the IoU for each class between predicted and ground truth masks, then averages across all classes.
- `kNN Accuracy` — range: [0, 1]
  - Accuracy of a non-parametric k-Nearest Neighbors classifier (k=20) using frozen representations.
- `K-means Accuracy` — range: [0, 1]
  - Clustering accuracy obtained by applying K-means to frozen representations and matching cluster labels to ground truth.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for binary classification tasks in sparse probing experiments.

## Input / output format

**Input**: Sentinel-2 multispectral images (12 bands) and/or Sentinel-1 radar images (VV, VH backscatter), typically resized to 264×264 for classification or cropped to 96×96 for segmentation.

**Output**: Class labels (multi-label for BigEarthNet, single-label for others), segmentation masks, or frozen feature vectors for probing.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task):
    if task == 'classification':
        if gold.ndim > 1:  # multi-label
            return mean_average_precision(gold, predictions)
        else:
            return accuracy(gold, predictions.argmax(axis=1))
    elif task == 'segmentation':
        return mean_intersection_over_union(gold, predictions)
    elif task == 'knn':
        return knn_accuracy(gold, predictions, k=20)
    elif task == 'clustering':
        return clustering_accuracy(gold, predictions)
    elif task == 'sparse_probing':
        return f1_score(gold, predictions)
```

## Common pitfalls

- Using the full training set instead of the 10% split for BigEarthNet and fMoW-Sentinel, which inflates performance and breaks fair comparison.
- Comparing models trained under different data splits or hyperparameter budgets, as originally reported results vary significantly.
- Ignoring distribution shift when finetuning on datasets like fMoW-Sentinel where the model was pre-trained on the same data, giving baselines an unfair advantage.

## Evidence (verbatim from paper)

> Table 1: Classification results on four benchmarks, under finetuning (FT), and frozen linear (LP) and nonlinear (MLP) probing. BigEarthNet (10%) mAP, fMoW-Sentinel (10%) Top 1 Acc., EuroSAT Top 1 Acc., Canadian Cropland Top 1 Acc.

## Citation

```bibtex
@misc{fuller2023croma,
  title={CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders},
  author={Fuller et al. (2023)},
  year={2023},
  note={arXiv:2311.00566}
}
```

- arXiv: 2311.00566