Rewardmap Eval

---
name: rewardmap-eval
description: Evaluates fine-grained visual reasoning and spatial understanding on structured domains like transit maps. It probes the model's ability to follow complex routes, count stops, and verify true/false statements about visual layouts, while also measuring generalization to broader spatial and chart reasoning benchmarks. Use when the user wants to benchmark on ReasonMap, ReasonMap-Plus, SEED-Bench-2-Plus, SpatialEval, V*Bench, HRBench, ChartQA, MMStar, or asks about evaluating this task. Reports Weighted Acc..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.02240
  bibtex_key: feng2025rewardmap
  confidence: high
---

# rewardmap-eval

> RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning — Feng et al. (2025) (arXiv:2510.02240, 2025)

## What this evaluates

Evaluates fine-grained visual reasoning and spatial understanding on structured domains like transit maps. It probes the model's ability to follow complex routes, count stops, and verify true/false statements about visual layouts, while also measuring generalization to broader spatial and chart reasoning benchmarks.

## Datasets

- **ReasonMap** — total 696; splits: test (-1)
- **ReasonMap-Plus** — total 2570; splits: test (-1)
- **SEED-Bench-2-Plus** — total ?; splits: test (-1)
- **SpatialEval** — total ?; splits: test (-1)
- **V*Bench** — total ?; splits: test (-1)
- **HRBench** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)

## Metrics

- `Weighted Acc.` **(primary)** — range: percent
  - Accuracy adjusted by a difficulty-aware weighting scheme applied to ReasonMap and ReasonMap-Plus test sets. Weights are assigned based on fine-grained difficulty annotations for each question.
- `Weighted Map Score` — range: other
  - A spatial reasoning metric scoring the correctness of route/path extraction on transit maps, adjusted by difficulty weights.
- `Accuracy` — range: percent
  - Standard exact-match or binary correctness percentage reported on SEED-Bench-2-Plus, SpatialEval, V*Bench, HRBench, ChartQA, and MMStar.

## Input / output format

**Input**: An image (e.g., transit map, chart, or visual scene) paired with a natural language question or instruction.

**Output**: A natural language text response containing the answer, route description, or true/false verdict.

## Scoring recipe

```python
def compute_weighted_acc(predictions, golds, weights):
    weighted_correct = sum(w for p, g, w in zip(predictions, golds, weights) if p.strip().lower() == g.strip().lower())
    total_weight = sum(weights)
    return (weighted_correct / total_weight) * 100

def compute_standard_acc(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Greedy decoding (temperature=0) is used exclusively for all evaluations, which may disadvantage models that perform better with sampling.
- ReasonMap/Plus metrics use difficulty-aware weighting, so raw accuracy scores cannot be directly compared to the unweighted percentages on other benchmarks.
- Models frequently hallucinate routes or confuse stops on complex maps, which simple accuracy metrics may not fully capture without qualitative inspection.

## Evidence (verbatim from paper)

> Table 1: Evaluations of reference models and fine-tuned models on ReasonMap and ReasonMap-Plus. ... Weighted Acc. | Weighted Map Score ... We first evaluate on the test sets of ReasonMap and ReasonMap-Plus, and we report the metrics adjusted by the difficulty-aware weighting scheme... All evaluations use greedy decoding (temperature=0).

## Citation

```bibtex
@misc{feng2025rewardmap,
  title={RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning},
  author={Feng et al. (2025)},
  year={2025},
  note={arXiv:2510.02240}
}
```

- arXiv: 2510.02240