P2g Visual Reasoning Eval

---
name: p2g-visual-reasoning-eval
description: Evaluates the visual reasoning and text-understanding capabilities of multimodal large language models (MLLMs) on high-resolution, text-rich, and general semantic images. It probes whether agent-augmented grounding improves answer accuracy compared to vanilla MLLMs and proprietary models like GPT-4V. Use when the user wants to benchmark on DocVQA, ChartVQA, GQA, SEED, MM-VET, MME, P2GB, or asks about evaluating this task. Reports VQA score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.19322
  bibtex_key: chen2024p2g
  confidence: high
---

# p2g-visual-reasoning-eval

> Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models — Chen et al. (2024) (arXiv:2403.19322, 2024)

## What this evaluates

Evaluates the visual reasoning and text-understanding capabilities of multimodal large language models (MLLMs) on high-resolution, text-rich, and general semantic images. It probes whether agent-augmented grounding improves answer accuracy compared to vanilla MLLMs and proprietary models like GPT-4V.

## Datasets

- **DocVQA** — total ?; splits: test (-1)
- **ChartVQA** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **MM-VET** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **P2GB** — total ?; splits: test (-1)

## Metrics

- `VQA score` **(primary)** — range: percent
  - Standard dataset-specific evaluation metric (typically exact match or fuzzy match accuracy) used for each visual question answering benchmark. Scores are reported as percentages in the tables.

## Input / output format

**Input**: High-resolution image paired with a natural language question or instruction.

**Output**: Natural language answer string, optionally accompanied by generated reasoning and bounding box coordinates for grounded objects/texts.

## Scoring recipe

```python
def compute_vqa_score(predictions, gold_answers):
    score = 0
    for pred, golds in zip(predictions, gold_answers):
        # Standard VQA evaluation: exact match or fuzzy match (e.g., 3-gram overlap)
        if normalize_answer(pred) in [normalize_answer(g) for g in golds]:
            score += 1
    return (score / len(predictions)) * 100
```

## Common pitfalls

- Benchmark scores are highly sensitive to input resolution and context window limits; high-res images may be truncated or downsampled.
- Text-rich benchmarks require external OCR agents; OCR errors or missed text directly degrade scores.
- Including relative position vectors of grounded boxes in the prompt significantly impacts performance on structured data like charts.

## Evidence (verbatim from paper)

> Table 5: Experimental results of P2G and baselines on visual reasoning benchmarks. The performance of P2G on visual reasoning benchmarks are presented in Table 6. On text-rich visual reasoning, P2G significantly outperform baselines, including the vanilla LLaVA, by more than doubled (3× on DocVQA, 2.4× on ChartVQA)... On general visual reasoning benchmarks, P2G also enjoys a consistent improvement over LLaVA and InstrtuctBLIP, demonstrating the superiority of P2G.

## Citation

```bibtex
@misc{chen2024p2g,
  title={Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.19322}
}
```

- arXiv: 2403.19322