Math Reward Eval

---
name: math-reward-eval
description: Evaluates multimodal and language-only models on mathematical reasoning, self-judgment/reward accuracy, and general multimodal capabilities. It measures how well a model can solve complex problems, verify its own answers, and generalize across diverse domains without external reward models. Use when the user wants to benchmark on MathVista, GSM8k, RewardBench2, VL-RewardBench, MMBench, MMStar, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22624
  bibtex_key: liu2025spark
  confidence: high
---

# math-reward-eval

> SPARK: Synergistic Policy And Reward Co-Evolving Framework — Liu et al. (2025) (arXiv:2509.22624, 2025)

## What this evaluates

Evaluates multimodal and language-only models on mathematical reasoning, self-judgment/reward accuracy, and general multimodal capabilities. It measures how well a model can solve complex problems, verify its own answers, and generalize across diverse domains without external reward models.

## Datasets

- **MathVista** — total ?; splits: test (-1)
- **GSM8k** — total ?; splits: test (-1)
- **RewardBench2** — total ?; splits: test (-1)
- **VL-RewardBench** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's predicted answer or correctness judgment matches the ground truth. Computed as (correct predictions / total instances) * 100.

## Input / output format

**Input**: Text and/or image prompts containing mathematical problems or general multimodal questions.

**Output**: Final answer string or a binary correctness judgment (correct/incorrect).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for p, g in zip(predictions, gold):
        if normalize_answer(p) == normalize_answer(g):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Reward and math benchmarks often require strict formatting for answer extraction; minor formatting differences (e.g., fractions vs decimals, extra whitespace) can cause false negatives if not normalized.
- Self-judgment accuracy evaluation requires the model to output both reasoning steps and a binary correctness label, which can be affected by chain-of-thought verbosity or inconsistent labeling formats.

## Evidence (verbatim from paper)

> We evaluate the model’s judgment ability by measuring its accuracy in determining whether its own answers are correct. Based on these judgments, we compute recall, precision, and F1 scores.

## Citation

```bibtex
@misc{liu2025spark,
  title={SPARK: Synergistic Policy And Reward Co-Evolving Framework},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.22624}
}
```

- arXiv: 2509.22624