Satellite Llp Eval

---
name: satellite-llp-eval
description: Evaluates the ability of lightweight deep learning models to predict fine-grained class proportions (e.g., vegetation density, population) from satellite image chips. The protocol measures how well models trained on coarse administrative-level label proportions can recover fine-grained spatial distributions, using both proportion regression and pixel-level segmentation accuracy. Use when the user wants to benchmark on esaworldcover, humanpop, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.12461
  bibtex_key: ramospollan2023onorbit
  confidence: high
---

# satellite-llp-eval

> On-orbit model training for satellite imagery with label proportions — Ramos-Pollán et al. (2023) (arXiv:2306.12461, 2023)

## What this evaluates

Evaluates the ability of lightweight deep learning models to predict fine-grained class proportions (e.g., vegetation density, population) from satellite image chips. The protocol measures how well models trained on coarse administrative-level label proportions can recover fine-grained spatial distributions, using both proportion regression and pixel-level segmentation accuracy.

## Datasets

- **esaworldcover** — total ?; splits: test (-1); repo https://github.com/rramosp/llpeo
- **humanpop** — total ?; splits: test (-1); repo https://github.com/rramosp/llpeo

## Metrics

- `MAE` **(primary)** — range: [0, 1]
  - Mean Absolute Error computed on the predicted label proportions per chip compared to the ground truth label proportions derived from pixel-level labels.
- `F1` — range: [0, 1]
  - Standard F1 score computed from the pixel-level segmentation predictions against ground truth segmentation maps.

## Input / output format

**Input**: Satellite image chips (geospatial raster data).

**Output**: Pixel-level segmentation maps (class predictions per pixel).

## Scoring recipe

```python
for each chip:
  gt_props = count_pixels_by_class(gt_map) / total_pixels
  pred_props = count_pixels_by_class(pred_map) / total_pixels
  mae = mean(abs(pred_props - gt_props))
  f1 = compute_f1_score(pred_map, gt_map)
return mae, f1
```

## Common pitfalls

- Models are trained using only coarse administrative-level label proportions, but evaluated on fine-grained chip-level proportions.
- Simpler models with fewer parameters consistently outperform larger segmentation models on this task, contrary to typical deep learning expectations.
- Evaluation combines both proportion regression (MAE) and pixel segmentation (F1), which may not always correlate.

## Evidence (verbatim from paper)

> Given the pixel level labels for each chip we compute the label proportions and compare both the segmentation model output (F1) and its proportions (MAE).

## Citation

```bibtex
@misc{ramospollan2023onorbit,
  title={On-orbit model training for satellite imagery with label proportions},
  author={Ramos-Pollán et al. (2023)},
  year={2023},
  note={arXiv:2306.12461}
}
```

- arXiv: 2306.12461