Kvg Bench Eval

---
name: kvg-bench-eval
description: Evaluates knowledge-intensive visual grounding (KVG), requiring models to combine domain-specific reasoning with fine-grained visual perception to locate specific entities in images containing multiple similar objects. Use when the user wants to benchmark on KVG-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12797
  bibtex_key: ma2025deepperception
  confidence: high
---

# kvg-bench-eval

> DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding — Ma et al. (2025) (arXiv:2503.12797, 2025)

## What this evaluates

Evaluates knowledge-intensive visual grounding (KVG), requiring models to combine domain-specific reasoning with fine-grained visual perception to locate specific entities in images containing multiple similar objects.

## Datasets

- **KVG-Bench** — total 1300; splits: test (1300); repo https://github.com/thunlp/DeepPerception

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of test cases where the predicted bounding box achieves an Intersection over Union (IoU) with the ground truth that meets or exceeds a threshold τ. Correct predictions are counted as 1, others as 0, then averaged.

## Input / output format

**Input**: An image containing multiple similar entities, accompanied by a natural language question/prompt specifying the target entity and often requiring domain knowledge to disambiguate.

**Output**: A structured response containing a reasoning chain enclosed in <think>...</think> tags, followed by the predicted bounding box coordinates enclosed in <answer>...</answer> tags.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_boxes, threshold=0.5):
    correct = 0
    for pred, gold in zip(predictions, gold_boxes):
        pred_box = extract_box_from_answer_tags(pred)
        iou = calculate_iou(pred_box, gold)
        if iou >= threshold:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models may output degenerate bounding boxes (e.g., [0,0,1000,1000]) to exploit weak rewards; the evaluation enforces a strict IoU threshold τ to prevent this.
- Failure to adhere to the <think>...</think> and <answer>...</answer> format tags leads to parsing errors and invalid predictions.
- Domain knowledge is required to distinguish visually similar objects; models relying solely on visual features without reasoning will fail.

## Evidence (verbatim from paper)

> DeepPerception introduces knowledge-intensive visual grounding (KVG), a task requiring fine-grained perception and domain-specific reasoning, to bridge the gap between MLLMs' knowledge and human-like visual cognition. ... enabling superior accuracy (+8.08%) and cross-domain generalization (+4.60%) over baseline methods on KVG-Bench, a manually curated benchmark spanning 10 domains with 1.3K test cases. The IoU reward evaluates the spatial alignment between predicted bounding boxes and ground-truth annotations. Given the ground-truth bounding box B and the predicted bounding box 	ilde{B}, the IoU reward is formally defined as: R_IoU = IoU(B, 	ilde{B}) if IoU >= τ else 0.

## Citation

```bibtex
@misc{ma2025deepperception,
  title={DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2503.12797}
}
```

- arXiv: 2503.12797