Geommbench Eval

---
name: geommbench-eval
description: Evaluates expert-level multimodal intelligence in geoscience and remote sensing by testing domain knowledge, perceptual grounding, and spatiotemporal reasoning across diverse sensors, disciplines, and task complexities. Use when the user wants to benchmark on GeoMMBench, or asks about evaluating this task. Reports Micro-averaged accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08896
  bibtex_key: xiao2026geommbench
  confidence: high
---

# geommbench-eval

> GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing — Xiao et al. (2026) (arXiv:2604.08896, 2026)

## What this evaluates

Evaluates expert-level multimodal intelligence in geoscience and remote sensing by testing domain knowledge, perceptual grounding, and spatiotemporal reasoning across diverse sensors, disciplines, and task complexities.

## Datasets

- **GeoMMBench** — total 1053; splits: val (-1), test (-1)

## Metrics

- `Micro-averaged accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions, calculated by dividing the number of correct predictions by the total number of questions. Invalid outputs are marked incorrect.

## Input / output format

**Input**: Image-based multiple-choice questions requiring identification, reasoning, or interpretation of geospatial and remote sensing data.

**Output**: Option letters (e.g., A, B, C, D) and/or numerical answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        extracted = extract_regex(pred)  # regex for option letters or numbers
        if extracted is None:
            continue  # invalid output marked incorrect
        if extracted == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models may fail on visual tasks despite possessing strong textual geospatial knowledge, indicating visual-linguistic misalignment rather than true knowledge gaps.
- Domain-specialized MLLMs fine-tuned only on perception tasks often underperform general-purpose models on broader reasoning benchmarks.
- Text-only baselines can achieve high scores on knowledge questions but drop significantly when visual grounding is required, highlighting modality-specific evaluation needs.

## Evidence (verbatim from paper)

> Model responses are evaluated through an automated rule-based pipeline using regular expressions to extract option letters and numerical answers, with invalid outputs marked as incorrect. Micro-averaged accuracy serves as the primary metric.

## Citation

```bibtex
@misc{xiao2026geommbench,
  title={GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing},
  author={Xiao et al. (2026)},
  year={2026},
  note={arXiv:2604.08896}
}
```

- arXiv: 2604.08896