Lae 1m Eval

---
name: lae-1m-eval
description: Evaluates open-vocabulary and closed-set object detection capabilities on remote sensing imagery. It probes a model's ability to detect novel Earth-based objects without prior training on them, as well as its efficiency when fine-tuned with limited labeled data. Use when the user wants to benchmark on LAE-1M, DIOR, DOTAv2.0, LAE-80C, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.09110
  bibtex_key: pan2024locate
  confidence: high
---

# lae-1m-eval

> Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community — Pan et al. (2024) (arXiv:2408.09110, 2024)

## What this evaluates

Evaluates open-vocabulary and closed-set object detection capabilities on remote sensing imagery. It probes a model's ability to detect novel Earth-based objects without prior training on them, as well as its efficiency when fine-tuned with limited labeled data.

## Datasets

- **LAE-1M** — total 1000000; splits: train (1000000); repo https://github.com/jaychempan/LAE-DINO
- **DIOR** — total ?; splits: test (-1)
- **DOTAv2.0** — total ?; splits: test (-1)
- **LAE-80C** — total ?; splits: test (-1)

## Metrics

- `mAP` **(primary)** — range: [0, 100] percent
  - Mean Average Precision computed across all evaluated classes. Standard COCO-style calculation averaging AP over multiple IoU thresholds or a single threshold depending on the benchmark configuration.
- `AP50` — range: [0, 100] percent
  - Average Precision calculated at an Intersection over Union (IoU) threshold of 0.5. Measures detection accuracy with relaxed localization requirements.
- `AP75` — range: [0, 100] percent
  - Average Precision calculated at an IoU threshold of 0.75. Measures detection accuracy with strict localization requirements.

## Input / output format

**Input**: Remote sensing satellite or aerial images. For open-vocabulary evaluation, text prompts or class names are provided as queries. For closed-set fine-tuning, images are paired with ground-truth bounding boxes and class labels.

**Output**: A list of predicted bounding boxes (coordinates) and associated class labels with confidence scores for each input image.

## Scoring recipe

```python
def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    # Sort predictions by confidence descending
    # Match each prediction to the highest IoU ground truth box
    # Classify as TP if IoU >= iou_thresh and box not yet matched, else FP
    # Compute precision-recall curve and interpolate AP
    return ap

def compute_mAP(predictions_per_class, ground_truth_per_class, iou_thresh=0.5):
    aps = [compute_ap(preds, gts, iou_thresh) for preds, gts in predictions_per_class.items()]
    return sum(aps) / len(aps) * 100
```

## Common pitfalls

- DOTAv2.0 results must strictly use horizontal bounding boxes, not rotated ones, as explicitly required by the authors.
- Open-vocabulary pre-training on natural scene datasets (e.g., O365, GoldG) performs poorly on remote sensing benchmarks due to a substantial domain gap; direct comparison without domain adaptation is misleading.
- LAE-1M construction uses class-dependent sampling rates (0.4 for most classes, 0.2 for Xview) to balance distribution, which affects training data composition and should be noted when reproducing.

## Evidence (verbatim from paper)

> The $mAP$, $AP_{50}$, and $AP_{75}$ are used as the evaluation metrics. Note that the results of DOTAv2.0 are all based on horizontal detection boxes for building a foundational location detector.

## Citation

```bibtex
@misc{pan2024locate,
  title={Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community},
  author={Pan et al. (2024)},
  year={2024},
  note={arXiv:2408.09110}
}
```

- arXiv: 2408.09110