Maestro Eval

---
name: maestro-eval
description: This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies. Use when the user wants to benchmark on TreeSatAI-TS, PASTIS-HD, FLAIR#2, FLAIR-HUB, or asks about evaluating this task. Reports weighted F1 score, mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.10894
  bibtex_key: labatie2025maestro
  confidence: high
---

# maestro-eval

> MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data — Labatie et al. (2025) (arXiv:2508.10894, 2025)

## What this evaluates

This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies.

## Datasets

- **TreeSatAI-TS** — total 50381; splits: train (-1), val (-1), test (-1)
- **PASTIS-HD** — total 433; splits: fold I (-1)
- **FLAIR#2** — total 77762; splits: split 1 (-1)
- **FLAIR-HUB** — total 241100; splits: split 1 (-1)

## Metrics

- `weighted F1 score` **(primary)** — range: [0, 100] percent
  - F1 score computed per class and averaged, weighted by the number of true instances per class. Calculated as 2 * (precision * recall) / (precision + recall + epsilon).
- `mIoU` **(primary)** — range: [0, 100] percent
  - Mean Intersection over Union across all semantic classes. Computed as the average of TP / (TP + FP + FN) for each class, where TP, FP, and FN are true positives, false positives, and false negatives.

## Input / output format

**Input**: Multimodal image tiles containing aerial imagery (RGB+NIR), Sentinel-1/2 time series, and elevation data (DSM/DEM), processed into fixed-size patches for token-based model input.

**Output**: Per-pixel or per-tile class predictions: 15 multi-label tree species classes for TreeSatAI-TS, or 12/15 semantic segmentation masks for PASTIS-HD, FLAIR#2, and FLAIR-HUB.

## Scoring recipe

```python
def compute_miou(preds, gold, num_classes):
    ious = []
    for c in range(num_classes):
        tp = np.sum((preds == c) & (gold == c))
        fp = np.sum((preds == c) & (gold != c))
        fn = np.sum((preds != c) & (gold == c))
        iou = tp / (tp + fp + fn + 1e-6)
        ious.append(iou)
    return np.mean(ious) * 100

def compute_weighted_f1(preds, gold, num_classes):
    f1s, weights = [], []
    for c in range(num_classes):
        tp = np.sum((preds == c) & (gold == c))
        fp = np.sum((preds == c) & (gold != c))
        fn = np.sum((preds != c) & (gold == c))
        prec = tp / (tp + fp + 1e-6)
        rec = tp / (tp + fn + 1e-6)
        f1 = 2 * prec * rec / (prec + rec + 1e-6)
        f1s.append(f1)
        weights.append(np.sum(gold == c))
    return np.average(f1s, weights=weights) * 100
```

## Common pitfalls

- Pre-training must strictly exclude test data; using the union of training and validation sets for self-supervised learning is required to prevent data leakage.
- Models must be evaluated on the exact dataset splits/folds specified (e.g., PASTIS-HD fold I, FLAIR split 1) to ensure comparability with reported baselines.
- Baseline foundation models require extensive hyperparameter tuning for fair comparison, as default settings may not reflect operational performance.

## Evidence (verbatim from paper)

> We report the weighted F1 score (%) on TreeSatAI-TS and the mIoU (%) on PASTIS-HD, FLAIR#2, and FLAIR-HUB.

## Citation

```bibtex
@misc{labatie2025maestro,
  title={MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data},
  author={Labatie et al. (2025)},
  year={2025},
  note={arXiv:2508.10894}
}
```

- arXiv: 2508.10894