Bigearthnet Txt Eval

---
name: bigearthnet-txt-eval
description: Evaluates vision-language models on remote sensing tasks including image captioning, binary visual question answering, multiple-choice questions, and referring expression/point detection. It probes the models' ability to understand multi-sensor (SAR + multispectral) and RGB Earth observation imagery, follow complex spatial instructions, and generate accurate land-use/land-cover descriptions or localized bounding boxes. Use when the user wants to benchmark on BigEarthNet.txt, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29630
  bibtex_key: herzog2026bigearthnettxt
  confidence: high
---

# bigearthnet-txt-eval

> BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation — Herzog et al. (2026) (arXiv:2603.29630, 2026)

## What this evaluates

Evaluates vision-language models on remote sensing tasks including image captioning, binary visual question answering, multiple-choice questions, and referring expression/point detection. It probes the models' ability to understand multi-sensor (SAR + multispectral) and RGB Earth observation imagery, follow complex spatial instructions, and generate accurate land-use/land-cover descriptions or localized bounding boxes.

## Datasets

- **BigEarthNet.txt** — total 464044; splits: benchmark (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions for binary VQA and MCQ tasks. Calculated as the number of exact matches between predicted and gold answers divided by total instances.
- `mIoU` — range: percent
  - Mean intersection-over-union between predicted and reference bounding boxes or segmentation masks for referring expression and point detection tasks.
- `BLEU-4` — range: percent
  - N-gram-based metric measuring the precision of 4-gram overlaps between generated captions and reference captions, with a brevity penalty.
- `CLAIR` — range: percent
  - LLM-based metric where a judge model (DeepSeek-R1-Distill-Qwen-32B) outputs a score between 0 and 100 indicating how likely the candidate caption describes the same image as the reference.

## Input / output format

**Input**: Multi-sensor (Sentinel-1 SAR + Sentinel-2 multispectral) or RGB (Sentinel-2 bands) Earth observation images paired with task-specific instructions (e.g., captioning prompts, binary/MCQ questions, referring expressions with or without a spatial point prior).

**Output**: Text responses (captions, yes/no answers, selected options, or bounding box coordinates) extracted to match the specified format; unambiguous answers are required for instruction-following tracking.

## Scoring recipe

```python
def evaluate(predictions, golds, task_type):
    # Extract answers even if they do not strictly adhere to the specified format
    preds_clean = extract_unambiguous_answers(predictions)
    if task_type in ('vqa', 'mcq'):
        return sum(1 for p, g in zip(preds_clean, golds) if p == g) / len(golds) * 100
    elif task_type == 'referring_detection':
        ious = [iou(p, g) for p, g in zip(preds_clean, golds)]
        mIoU = sum(ious) / len(ious) * 100
        acc_k = {k: sum(1 for i in ious if i >= k/100) / len(ious) * 100 for k in [25, 50, 75, 90]}
        return mIoU, acc_k
    elif task_type == 'captioning':
        return compute_ngram_embedding_metrics(preds_clean, golds)
    elif task_type == 'captioning_clair':
        return llm_judge_score(preds_clean, golds)
```

## Common pitfalls

- Models accepting multispectral/multi-sensor inputs often perform worse than RGB-only versions because they were pre-trained/fine-tuned on RGB data and lack inference-time adaptation to exploit extra spectral bands.
- Instruction-following is a major bottleneck; many models fail to consistently adhere to MCQ or binary VQA formats, requiring careful post-hoc answer extraction to compute accuracy.
- Referring expression detection without a point prior is significantly harder than with a point prior, as the latter reduces the task to local boundary estimation rather than open-vocabulary instance search.

## Evidence (verbatim from paper)

> Reported metrics for captioning: BLEU-4, binary VQA (visual question answering): accuracy, MCQ (multiple-choice question): accuracy, and referring expression detection: mIoU (mean intersection-over-union). All results in percent (%).

## Citation

```bibtex
@misc{herzog2026bigearthnettxt,
  title={BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation},
  author={Herzog et al. (2026)},
  year={2026},
  note={arXiv:2603.29630}
}
```

- arXiv: 2603.29630