Geobench Eval

---
name: geobench-eval
description: Evaluates the transferability and semantic grounding of remote sensing foundation models on image-level classification and pixel-level semantic segmentation tasks across multiple geospatial benchmarks. Use when the user wants to benchmark on GeoBench, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10591
  bibtex_key: hasan2026geomeld
  confidence: high
---

# geobench-eval

> GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing — Hasan et al. (2026) (arXiv:2604.10591, 2026)

## What this evaluates

Evaluates the transferability and semantic grounding of remote sensing foundation models on image-level classification and pixel-level semantic segmentation tasks across multiple geospatial benchmarks.

## Datasets

- **GeoBench** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, computed per class and averaged.
- `Accuracy` — range: [0, 1]
  - Ratio of correctly predicted samples to total samples.
- `IoU` — range: [0, 1]
  - Intersection over union between predicted and ground-truth segmentation masks.
- `Recall@5` — range: [0, 1]
  - Fraction of relevant items retrieved within the top-5 ranked results for cross-modal retrieval.

## Input / output format

**Input**: Spatially aligned 128×128 remote sensing image tiles (primarily Sentinel-2 12-band) paired with task-specific ground-truth labels (class vectors for classification, pixel masks for segmentation).

**Output**: Predicted class labels or probabilities for classification tasks; predicted pixel-wise segmentation masks for segmentation tasks.

## Scoring recipe

```python
def compute_metrics(preds, golds, task_type):
    if task_type == 'classification':
        acc = (preds == golds).mean()
        f1 = f1_score(golds, preds, average='macro')
        return {'Accuracy': acc, 'F1': f1}
    elif task_type == 'segmentation':
        ious = []
        for p, g in zip(preds, golds):
            inter = np.logical_and(p, g).sum()
            union = np.logical_or(p, g).sum()
            ious.append(inter / (union + 1e-6))
        return {'IoU': np.mean(ious)}
    elif task_type == 'retrieval':
        # Recall@5: fraction of queries where at least one relevant item is in top-5
        pass
```

## Common pitfalls

- Confusing linear probing (frozen encoder, 50 epochs) with full fine-tuning (all parameters updated, 50 epochs for classification, 100 for segmentation).
- Not specifying macro vs. micro averaging for F1 score, which can significantly change reported values.
- Mixing up pretraining multi-modal inputs (S2, SAR, DEM) with downstream task inputs, which are task-specific.

## Evidence (verbatim from paper)

> Table 2: Downstream evaluation results on GeoBench dataset. FT = full fine-tuning, LP = linear probing. | Pretrain data | BigEarthNet20k (F1↑)FT/LP | So2Sat20k (Acc.↑)FT/LP | Cashew1k (IoU↑)FT | SAcrop3k (IoU↑)FT |

## Citation

```bibtex
@misc{hasan2026geomeld,
  title={GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing},
  author={Hasan et al. (2026)},
  year={2026},
  note={arXiv:2604.10591}
}
```

- arXiv: 2604.10591