Curr Reft Eval

---
name: curr-reft-eval
description: Evaluates the out-of-domain generalization and reasoning capabilities of vision-language models across visual detection, classification, and multimodal mathematical reasoning tasks, alongside standard multimodal benchmarks. Use when the user wants to benchmark on RefCOCO, RefGTA, Pascal-VOC, Math360K, CLEVER-70k-Counting, MathVista, MATH, AI2D, MMBench, MMVet, OCRBench, LLaVABench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07065
  bibtex_key: deng2025currreft
  confidence: high
---

# curr-reft-eval

> Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning — Deng et al. (2025) (arXiv:2503.07065, 2025)

## What this evaluates

Evaluates the out-of-domain generalization and reasoning capabilities of vision-language models across visual detection, classification, and multimodal mathematical reasoning tasks, alongside standard multimodal benchmarks.

## Datasets

- **RefCOCO** — total 4000; splits: train (3000), test (1000)
- **RefGTA** — total 1000; splits: test (1000)
- **Pascal-VOC** — total 1000; splits: test (1000)
- **Math360K** — total 4000; splits: train (3000), test (1000)
- **CLEVER-70k-Counting** — total 500; splits: test (500)
- **MathVista** — total 6141; splits: test (6141)
- **MATH** — total 12000; splits: test (12000)
- **AI2D** — total 5000; splits: test (5000)
- **MMBench** — total 3000; splits: test (3000)
- **MMVet** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **LLaVABench** — total 24000; splits: test (24000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Correct predictions divided by total test samples. For detection, correctness requires IoU > 0.5 between predicted and ground truth boxes. For classification, exact label match is required. For math, final answer extraction is used.

## Input / output format

**Input**: Image(s) paired with text prompts or questions for tasks including object localization, visual categorization, and multimodal math reasoning.

**Output**: Text responses containing bounding box coordinates (for detection), class labels (for classification), or step-by-step reasoning and final answers (for math).

## Scoring recipe

```python
def compute_accuracy(predictions, golds, task_type):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if task_type == 'detection':
            correct += 1 if iou(pred, gold) > 0.5 else 0
        elif task_type == 'classification':
            correct += 1 if pred == gold else 0
        elif task_type == 'math':
            correct += 1 if extract_final_answer(pred) == gold else 0
    return correct / len(golds) * 100
```

## Common pitfalls

- IoU threshold of 0.5 is strictly required for detection correctness, not just label matching.
- MMVet uses an LLM-based evaluator for unified scoring across 6 sub-tasks, which may introduce evaluator bias.
- Out-of-domain splits (e.g., RefGTA, Pascal-VOC, CLEVER) are evaluated separately from in-domain splits to measure generalization.

## Evidence (verbatim from paper)

> We use accuracy as unified evaluation metric, defined as correct predictions over total test samples. For detection, a prediction is correct if the IoU between predicted and ground truth boxes exceeds 0.5. In classification, predictions matching ground truth labels are considered correct.

## Citation

```bibtex
@misc{deng2025currreft,
  title={Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2503.07065}
}
```

- arXiv: 2503.07065