Flairhub Eval

---
name: flairhub-eval
description: Evaluates semantic segmentation models for fine-grained land cover classification and crop type mapping using multi-sensor remote sensing imagery. It probes the model's ability to fuse spatial, spectral, and temporal modalities (aerial RGBI, SPOT, Sentinel-1/2, DEM) for pixel-level prediction at 20 cm resolution. Use when the user wants to benchmark on FLAIR-HUB, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.07080
  bibtex_key: [REDACTED]
  confidence: high
---

# flairhub-eval

> FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping — Garioud et al. (2025) (arXiv:2506.07080, 2025)

## What this evaluates

Evaluates semantic segmentation models for fine-grained land cover classification and crop type mapping using multi-sensor remote sensing imagery. It probes the model's ability to fuse spatial, spectral, and temporal modalities (aerial RGBI, SPOT, Sentinel-1/2, DEM) for pixel-level prediction at 20 cm resolution.

## Datasets

- **FLAIR-HUB** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union computed per class and averaged. For land-cover, calculated over 15 valid classes (excluding ill-defined ones). For crop-type, calculated over classes present in the test set (excluding rice and other oilseed crops).
- `O.A.` — range: [0, 1]
  - Overall Accuracy, defined as the ratio of correctly predicted pixels to the total number of pixels across all classes.

## Input / output format

**Input**: Multi-channel remote sensing imagery (AERIAL_RGBI or SPOT_RGBI with Infrared, Red, Green channels; DEM_ELEV; AERIAL_RLT-PAN) normalized using dataset statistics, aligned spatially and temporally for pixel-level semantic segmentation.

**Output**: Per-pixel class predictions for land cover (15 classes) or crop type classification.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, valid_classes):
    ious = []
    correct = 0
    total = 0
    for cls in valid_classes:
        pred_cls = (predictions == cls)
        gt_cls = (ground_truth == cls)
        intersection = np.sum(pred_cls & gt_cls)
        union = np.sum(pred_cls | gt_cls)
        ious.append(intersection / union if union > 0 else 1.0)
        correct += np.sum(pred_cls & gt_cls)
        total += np.sum(gt_cls)
    mIoU = np.mean(ious)
    OA = correct / total
    return mIoU, OA
```

## Common pitfalls

- mIoU is computed only over a subset of classes (15 for land cover, excluding ill-defined ones; crop-type excludes rice and other oilseed crops), not all available classes.
- Input channels for aerial/SPOT imagery are restricted to Infrared, Red, and Green (3 channels), not the full 4-channel RGBI.
- Normalization statistics are computed over the combined TRAIN and VAL partitions, not the full dataset or test set alone.

## Evidence (verbatim from paper)

> Metric: The performance of the semantic segmentation models is evaluated using the mean Intersection over Union (mIoU) and Overall Accuracy (O.A.) metrics. For the land-cover task, we exclude the ill-defined classes (see Table[VI]) and thus evaluate the results over the remaining 15 classes. For the LPIS crop-type task, two classes, rice and other oilseed crops, are absent from the test set, so mIoU is computed over the remaining classes.

## Citation

```bibtex
@misc{garioud2025flairhub,
  title={FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping},
  author={Garioud et al. (2025)},
  year={2025},
  note={arXiv:2506.07080}
}
```

- arXiv: 2506.07080